Sep 17 17:33:46.854450 ip-10-0-134-52 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:33:50.576349 ip-10-0-134-52 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:50.576349 ip-10-0-134-52 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:33:50.576349 ip-10-0-134-52 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:50.576349 ip-10-0-134-52 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:33:50.576349 ip-10-0-134-52 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:50.577138 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.576432    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:33:50.580660 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580642    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.580660 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580658    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580664    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580667    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580670    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580674    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580677    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580680    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580682    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580685    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580688    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580690    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580694    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580698    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580701    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580704    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580706    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580709    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580711    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580720    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580723    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.580736 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580725    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580728    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580731    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580733    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580736    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580740    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580743    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580746    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580749    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580752    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580755    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580758    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580761    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580763    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580766    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580769    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580771    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580774    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580777    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580779    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.581245 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580782    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580786    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580789    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580793    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580796    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580799    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580802    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580805    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580808    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580810    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580813    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580816    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580818    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580822    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580825    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580828    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580831    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580834    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580837    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.581740 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580839    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580842    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580844    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580847    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580849    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580852    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580855    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580857    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580860    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580863    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580866    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580869    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580871    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580874    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580877    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580879    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580883    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580885    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580888    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.582225 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580891    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580893    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580896    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580899    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580901    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580904    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580906    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580910    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580915    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580919    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580922    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580925    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580928    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580931    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580934    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580936    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580939    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580942    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580944    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.580947    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.582710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581653    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581661    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581664    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581667    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581669    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581672    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581675    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581677    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581680    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581684    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581686    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581689    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581694    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581698    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581701    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581704    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581707    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581710    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581712    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581715    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.583218 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581719    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581723    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581725    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581728    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581733    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581736    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581739    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581743    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581746    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581748    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581751    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581754    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581757    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581759    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581762    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581765    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581767    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581770    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581773    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.583730 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581776    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581778    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581781    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581785    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581787    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581790    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581792    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581795    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581798    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581800    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581803    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581805    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581808    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581811    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581814    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581816    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581820    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581822    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581825    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.584215 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581827    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581830    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581834    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581837    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581839    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581842    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581844    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581846    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581849    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581852    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581855    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581857    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581860    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581862    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581865    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581867    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581870    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581873    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581875    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581878    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.584690 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581881    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581883    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581887    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581890    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581892    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581895    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581898    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581900    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581903    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581906    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581909    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581911    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581914    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581916    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581919    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581921    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581924    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581926    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581929    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581931    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.585233 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.581934    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582332    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582338    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582341    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582344    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582347    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582350    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582353    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582355    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582367    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582370    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582373    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582376    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582379    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582381    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582384    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582387    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582390    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582393    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.585737 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582396    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582398    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582403    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582407    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582410    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582412    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582415    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582417    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582420    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582422    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582427    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582430    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582432    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582435    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582437    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582440    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582442    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582445    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582448    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582450    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.586224 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582453    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582455    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582458    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582466    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582469    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582471    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582474    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582477    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582479    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582482    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582485    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582487    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582490    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582492    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582497    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582500    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582503    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582507    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582510    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582513    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.586743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582515    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582518    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582522    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582525    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582528    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582530    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582533    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582535    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582538    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582540    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582543    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582547    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582550    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582553    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582555    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582558    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582561    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582564    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582566    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.587283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582569    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582572    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582574    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582577    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582581    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582583    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582586    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582589    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582592    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582594    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582597    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582599    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582602    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582604    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582607    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582614    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582617    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582620    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582622    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582625    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.587776 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582628    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.582630    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582704    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582738    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582745    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582750    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582754    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582757    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582761    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582765    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582770    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582773    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582777    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582780    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582783    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582786    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582790    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582792    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582796    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582799    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582802    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582805    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582808    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582811    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:33:50.588271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582815    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582818    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582821    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582826    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582829    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582832    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582835    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582838    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582841    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582844    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582853    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582856    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582859    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582862    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582865    2575 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582868    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582873    2575 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582876    2575 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582878    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582882    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582886    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582889    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582895    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582898    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582901    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:33:50.588848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582904    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582907    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582910    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582913    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582916    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582919    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582922    2575 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582925    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582928    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582931    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582937    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582940    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582943    2575 flags.go:64] FLAG: --help="false"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582946    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582949    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582952    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582955    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582959    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582962    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582965    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582968    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582971    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582974    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:33:50.589466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582977    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582980    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582983    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582986    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582989    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582992    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582996    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.582999    2575 flags.go:64] FLAG: --lock-file=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583002    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583005    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583008    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583014    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583016    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583019    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583022    2575 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583025    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583029    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583032    2575 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583034    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583040    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583043    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583047    2575 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583050    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583053    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583056    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:33:50.590174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583059    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583062    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583065    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583068    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583075    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583078    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583081    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583084    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583087    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583107    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583110    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583113    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583116    2575 flags.go:64] FLAG: --port="10250"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583120    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583124    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09f18ec93f84437ee"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583127    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583130    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583133    2575 flags.go:64] FLAG: --register-node="true"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583136    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583139    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583142    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583145    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583148    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583152    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583155    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:33:50.591288 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583158    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583163    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583166    2575 flags.go:64] FLAG: --runonce="false"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583169    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583172    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583175    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583178    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583181    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583184    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583187    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583190    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583193    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583196    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583199    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583202    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583205    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583208    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583211    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583216    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583219    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583225    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583231    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583234    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583237    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583240    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:33:50.592440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583243    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583246    2575 flags.go:64] FLAG: --v="2"
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583250    2575 flags.go:64] FLAG: --version="false"
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583254    2575 flags.go:64] FLAG: --vmodule=""
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583257    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.583261    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583390    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583394    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583399    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583402    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583405    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583408    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583411    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583414    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583417    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583419    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583422    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583425    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583427    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583430    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583433    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.593562 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583436    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583440    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583444    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583447    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583450    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583453    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583455    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583461    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583464    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583466    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583469    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583472    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583474    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583477    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583480    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583482    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583485    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583488    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583490    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.594493 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583494    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583496    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583500    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583502    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583505    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583508    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583511    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583514    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583517    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583519    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583522    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583525    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583527    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583530    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583533    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583535    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583538    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583541    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583543    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583546    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.595033 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583550    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583553    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583556    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583559    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583562    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583565    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583567    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583570    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583574    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583578    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583581    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583585    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583589    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583592    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583596    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583599    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583602    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583605    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583607    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.596013 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583610    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583613    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583616    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583618    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583621    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583623    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583626    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583629    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583631    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583634    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583637    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583639    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583642    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583646    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583649    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583653    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583655    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583658    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583661    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583664    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.596583 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583666    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583670    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583672    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583675    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583678    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.583681    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.584331    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.592871    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.592897    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.592925    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.592938    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593032    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593040    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593045    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593050    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.597160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593055    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593059    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593064    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593069    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593073    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593078    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593082    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593086    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593090    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593121    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593126    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593130    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593133    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593138    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593142    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593146    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593151    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593155    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593159    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.597542 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593166    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593172    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593176    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593181    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593186    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593190    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593208    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593212    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593217    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593221    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593226    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593230    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593234    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593239    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593243    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593248    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593252    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593256    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593261    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593268    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.598246 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593273    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593278    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593283    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593287    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593291    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593295    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593299    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593303    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593307    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593311    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593316    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593320    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593324    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593328    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593333    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593337    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593341    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593345    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593350    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593363    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.598890 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593368    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593372    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593376    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593381    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593386    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593392    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593398    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593404    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593409    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593413    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593417    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593421    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593425    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593430    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593434    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593438    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593442    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593446    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593451    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.599434 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593455    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593459    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593463    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593468    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593472    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593476    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593481    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593485    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593490    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593494    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593498    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593511    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593515    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593527    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593532    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593536    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.599948 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593540    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.593549    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593779    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593788    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593793    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593797    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593802    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593807    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593812    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593816    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593821    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593825    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593829    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593833    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:50.600527 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593837    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593843    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593848    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593853    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593857    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593861    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593865    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593870    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593874    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593878    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593882    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593886    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593890    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593895    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593900    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593904    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593941    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593949    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593955    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593960    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:50.601090 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593964    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593969    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593973    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593978    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593982    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593986    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593990    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593994    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.593998    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594005    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594011    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594016    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594020    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594025    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594029    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594034    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594038    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594042    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594046    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594050    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:50.601841 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594055    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594060    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594064    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594069    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594073    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594078    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594082    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594086    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594090    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594125    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594130    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594134    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594138    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594142    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594146    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594150    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594155    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594159    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594164    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594168    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:50.602496 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594172    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594178    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594183    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594188    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594193    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594198    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594202    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594206    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594211    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594215    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594219    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594223    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594228    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594232    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594237    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594241    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594245    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594250    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594254    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594258    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:50.603036 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594262    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594267    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594282    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594287    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594291    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594295    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:50.594299    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.594307    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.594481    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.598683    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.599506    2575 server.go:1019] "Starting client certificate rotation"
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.599611    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.599989    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:50.603727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.603707    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:50.604919 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.604899    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:50.618637 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.618617    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:33:50.620198 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.620186    2575 log.go:25] "Validated CRI v1 image API"
Sep 17 17:33:50.621708 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.621679    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:33:50.626737 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.626712    2575 fs.go:135] Filesystem UUIDs: map[74ae4606-a352-4663-a22a-559c0ba10395:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a02c3215-f1e5-4cd1-9de1-cc933dba2f7c:/dev/nvme0n1p3]
Sep 17 17:33:50.626737 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.626730    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:33:50.632280 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.632181    2575 manager.go:216] Machine: {Timestamp:2026-09-17 17:33:50.631703136 +0000 UTC m=+1.998835007 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107556 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9e6f3f08013a2ea0c7545e94884a SystemUUID:ec2a9e6f-3f08-013a-2ea0-c7545e94884a BootID:5623e852-0832-454b-a244-a15202cc5ee0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:03:74:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:03:74:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:50:15:97:09:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:33:50.632280 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.632274    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:33:50.632390 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.632372    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:33:50.634246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.634225    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:33:50.634380 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.634248    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:33:50.635696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.635685    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:33:50.635730 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.635699    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:33:50.635730 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.635707    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:50.635730 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.635717    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:50.636782 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.636767    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:33:50.638091 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.638070    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:50.639364 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639352    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:50.639481 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639472    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:33:50.639514 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639499    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:33:50.639514 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639508    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:33:50.639570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639525    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:33:50.639570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639536    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:33:50.639570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639543    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:33:50.639848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639835    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:50.639914 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.639853    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:50.645262 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.645235    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:33:50.647408 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.647393    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:33:50.647469 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.647424    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:33:50.648332 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648319    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648337    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648343    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648348    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648356    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648361    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:33:50.648374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648374    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648380    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648386    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648392    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648400    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648406    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648428    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:33:50.648524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.648433    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:33:50.650301 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.650283    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:33:50.650352 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.650339    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:33:50.650437 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.650417    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:33:50.652243 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.652230    2575 server.go:1268] "Started kubelet"
Sep 17 17:33:50.652336 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.652296    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:33:50.652373 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.652357    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:33:50.652680 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.652654    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:33:50.652904 ip-10-0-134-52 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:33:50.653782 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.653374    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:33:50.654251 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.654239    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:33:50.658658 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.658638    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:33:50.659052 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659035    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:33:50.659052 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659047    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:50.659190 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.659161    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:50.659587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659568    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:33:50.659587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659587    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:33:50.659729 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659619    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:33:50.659729 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659716    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:33:50.659729 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659724    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:33:50.659858 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659850    2575 factory.go:55] Registering systemd factory
Sep 17 17:33:50.659907 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.659860    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:33:50.660056 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660038    2575 factory.go:153] Registering CRI-O factory
Sep 17 17:33:50.660056 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660056    2575 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:33:50.660196 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660120    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:33:50.660196 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660139    2575 factory.go:103] Registering Raw factory
Sep 17 17:33:50.660196 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660161    2575 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:33:50.660556 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.660544    2575 manager.go:349] Starting recovery of all containers
Sep 17 17:33:50.673410 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.673381    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:33:50.673518 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.673499    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:33:50.676774 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.676741    2575 manager.go:354] Recovery completed
Sep 17 17:33:50.678238 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.677389    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-52.ec2.internal.18d62c22f977494a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-52.ec2.internal,UID:ip-10-0-134-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-52.ec2.internal,},FirstTimestamp:2026-09-17 17:33:50.652205386 +0000 UTC m=+2.019337257,LastTimestamp:2026-09-17 17:33:50.652205386 +0000 UTC m=+2.019337257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-52.ec2.internal,}"
Sep 17 17:33:50.681256 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.681242    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.683077 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683053    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.683161 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683086    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.683161 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683110    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.684010 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683787    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:33:50.684010 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683800    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:33:50.684010 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.683818    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:50.684483 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.684463    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.685139 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685125    2575 policy_none.go:49] "None policy: Start"
Sep 17 17:33:50.685225 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685160    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:33:50.685225 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685176    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:33:50.685322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685260    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.685322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685283    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.685322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.685292    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.686173 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.686159    2575 policy_none.go:47] "Start"
Sep 17 17:33:50.687497 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.687479    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-th7v6"
Sep 17 17:33:50.692356 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.692300    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-52.ec2.internal.18d62c22fb4e5212  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-52.ec2.internal,UID:ip-10-0-134-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-52.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-52.ec2.internal,},FirstTimestamp:2026-09-17 17:33:50.68307509 +0000 UTC m=+2.050206962,LastTimestamp:2026-09-17 17:33:50.68307509 +0000 UTC m=+2.050206962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-52.ec2.internal,}"
Sep 17 17:33:50.695346 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.695290    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-52.ec2.internal.18d62c22fb4e90a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-52.ec2.internal,UID:ip-10-0-134-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-52.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-52.ec2.internal,},FirstTimestamp:2026-09-17 17:33:50.683091104 +0000 UTC m=+2.050222976,LastTimestamp:2026-09-17 17:33:50.683091104 +0000 UTC m=+2.050222976,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-52.ec2.internal,}"
Sep 17 17:33:50.696908 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.696893    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-th7v6"
Sep 17 17:33:50.702375 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.702319    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-52.ec2.internal.18d62c22fb4eed37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-52.ec2.internal,UID:ip-10-0-134-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-52.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-52.ec2.internal,},FirstTimestamp:2026-09-17 17:33:50.683114807 +0000 UTC m=+2.050246680,LastTimestamp:2026-09-17 17:33:50.683114807 +0000 UTC m=+2.050246680,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-52.ec2.internal,}"
Sep 17 17:33:50.726689 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.726677    2575 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.726711    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.726720    2575 server.go:87] "Starting device plugin registration server"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.726925    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.726947    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.727033    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.727114    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.727122    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.727580    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:33:50.735403 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.727628    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:50.770796 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.770776    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:33:50.771830 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.771812    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:33:50.771907 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.771862    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:33:50.771907 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.771880    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:33:50.772006 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.771912    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:33:50.773660 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.773643    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:50.827839 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.827788    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.828531 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.828515    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.828585 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.828544    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.828585 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.828554    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.828585 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.828573    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.835346 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.835331    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.835405 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.835352    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-52.ec2.internal\": node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:50.859219 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.859198    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:50.872802 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.872778    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"]
Sep 17 17:33:50.872859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.872831    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.873538 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.873524    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.873613 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.873549    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.873613 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.873560    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.874697 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.874685    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.874800 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.874786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.874845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.874815    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.875366 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875350    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.875443 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875351    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.875443 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875401    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.875443 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875415    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.875443 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875378    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.875573 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.875454    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.876726 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.876710    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.876774 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.876747    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:50.877432 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.877418    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:50.877493 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.877440    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:50.877493 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.877451    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:50.911638 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.911618    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-52.ec2.internal\" not found" node="ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.915907 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.915893    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-52.ec2.internal\" not found" node="ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.959503 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:50.959486    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:50.962725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.962712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afdef641cffa9a246bda35ec2b888902-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"afdef641cffa9a246bda35ec2b888902\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.962794 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.962733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:50.962794 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:50.962752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.059638 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.059603    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.063866 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afdef641cffa9a246bda35ec2b888902-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"afdef641cffa9a246bda35ec2b888902\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.063950 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.063950 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.063950 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afdef641cffa9a246bda35ec2b888902-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"afdef641cffa9a246bda35ec2b888902\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.064071 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.064071 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.063996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5330a9d330e353403bdebed08bc7f829-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"5330a9d330e353403bdebed08bc7f829\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.160359 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.160308    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.214647 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.214623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.219625 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.219605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.261150 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.261130    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.361632 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.361609    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.462135 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.462071    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.562681 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.562648    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.599978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.599955    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:33:51.600352 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.600070    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:33:51.662986 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.662966    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.696329 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.696308    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:51.697657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.697642    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:51.697748 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.697671    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:51.697748 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.697687    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:51.700819 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.700786    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:28:50 +0000 UTC" deadline="2028-07-01 18:42:51.901292012 +0000 UTC"
Sep 17 17:33:51.700819 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.700812    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15673h9m0.200482763s"
Sep 17 17:33:51.764128 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:51.763928    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found"
Sep 17 17:33:51.781841 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.781825    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:51.799547 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:51.799516    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdef641cffa9a246bda35ec2b888902.slice/crio-e03d1506cd6be4874fbb6c501b033770641979039542e1d48f44f14618811329 WatchSource:0}: Error finding container e03d1506cd6be4874fbb6c501b033770641979039542e1d48f44f14618811329: Status 404 returned error can't find the container with id e03d1506cd6be4874fbb6c501b033770641979039542e1d48f44f14618811329
Sep 17 17:33:51.799996 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:51.799983    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5330a9d330e353403bdebed08bc7f829.slice/crio-03157de71b28e2a5480dbb6f0fd8df44b0b07de7c7ba9cf4ab67aa2bc2e74137 WatchSource:0}: Error finding container 03157de71b28e2a5480dbb6f0fd8df44b0b07de7c7ba9cf4ab67aa2bc2e74137: Status 404 returned error can't find the container with id 03157de71b28e2a5480dbb6f0fd8df44b0b07de7c7ba9cf4ab67aa2bc2e74137
Sep 17 17:33:51.805230 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.805216    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:33:51.841726 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.841706    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:51.859321 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.859303    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.875432 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.875403    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:51.876823 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.876810    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"
Sep 17 17:33:51.886080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:51.886063    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:52.231260 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.230962    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:52.452877 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.452838    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:52.640722 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.640659    2575 apiserver.go:52] "Watching apiserver"
Sep 17 17:33:52.647470 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.647389    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:33:52.648080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.647759    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7","openshift-cluster-node-tuning-operator/tuned-8lgck","openshift-image-registry/node-ca-c8n74","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal","openshift-network-diagnostics/network-check-target-vzdrl","openshift-network-operator/iptables-alerter-gz8xp","openshift-ovn-kubernetes/ovnkube-node-zbx6h","kube-system/global-pull-secret-syncer-fxgnf","kube-system/konnectivity-agent-f5d72","openshift-dns/node-resolver-xnbdh","openshift-multus/multus-89kdm","openshift-multus/multus-additional-cni-plugins-hgh7c","openshift-multus/network-metrics-daemon-twm78"]
Sep 17 17:33:52.649832 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.649596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.650745 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.650718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.651859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.651840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.652405 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.652364    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:33:52.652405 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.652386    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:33:52.652543 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.652415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lslh7\""
Sep 17 17:33:52.653121 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.653089    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:52.653205 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.653181    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:33:52.653488 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.653465    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.653666 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.653652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6dzlz\""
Sep 17 17:33:52.653723 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.653669    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:33:52.654027 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.654011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.654438 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.654417    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.654855 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.654837    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:33:52.655001 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.654982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:33:52.655043 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.655019    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.655506 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.655189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:33:52.655506 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.655205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gvdb9\""
Sep 17 17:33:52.655506 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.655321    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.656618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.656602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.656720 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.656668    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:33:52.657602 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.657582    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.657799 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.657762    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.657799 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.657764    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7s95z\""
Sep 17 17:33:52.657962 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.657823    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:33:52.658855 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.658369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.659478 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.659459    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:52.659897 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.659864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.661206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.660682    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.661206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.660871    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.661206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.660883    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpmfh\""
Sep 17 17:33:52.661581 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.661563    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.662051 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.661932    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:33:52.662283 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.662265    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.662362 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.662285    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgp6d\""
Sep 17 17:33:52.662619 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.662604    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.662738 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.662720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.663817 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.663798    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.664215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.664056    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jvshz\""
Sep 17 17:33:52.664215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.664194    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.664390 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.664371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.665813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.665624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mjm9f\""
Sep 17 17:33:52.665813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.665689    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:33:52.665954 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.665914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.666007 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.665986    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:33:52.668166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.667954    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:33:52.668166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.668146    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:33:52.668299 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.668240    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:33:52.670472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.668857    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9w5b\""
Sep 17 17:33:52.670472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.669285    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:33:52.670472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.669662    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:33:52.670472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.669773    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:33:52.671495 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-modprobe-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.671587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.671587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-sys\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.671587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-etc-selinux\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.671741 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-lib-modules\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.671741 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.671839 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-iptables-alerter-script\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.671839 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.671839 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-run\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.671977 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.671918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-tuned\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.672076 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-registration-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.672076 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0616c07-ca03-4208-a925-cba77fe73ec1-host\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.672203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2024083-28b8-4e85-8dab-49e3b84bc155-konnectivity-ca\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.672203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qqj\" (UniqueName: \"kubernetes.io/projected/b8c30850-babf-4c71-8e02-ea922ec6732a-kube-api-access-69qqj\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.672300 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-device-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.672300 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whn2j\" (UniqueName: \"kubernetes.io/projected/56037b3a-b283-43e7-aad2-f0e19394874b-kube-api-access-whn2j\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.672300 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.672440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-sys-fs\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.672440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bnj\" (UniqueName: \"kubernetes.io/projected/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kube-api-access-b4bnj\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.672440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56037b3a-b283-43e7-aad2-f0e19394874b-hosts-file\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.672580 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672467    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56037b3a-b283-43e7-aad2-f0e19394874b-tmp-dir\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.672580 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-var-lib-kubelet\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.672675 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-dbus\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.672675 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2024083-28b8-4e85-8dab-49e3b84bc155-agent-certs\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.672771 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-cnibin\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.672771 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-conf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.672864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-systemd\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.672864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-host\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.672864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0616c07-ca03-4208-a925-cba77fe73ec1-serviceca\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.673003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-kubelet-config\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.673003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.672972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.673176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.673176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysconfig\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.673176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-tmp\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.673323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673199    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxnf\" (UniqueName: \"kubernetes.io/projected/de37802d-fd4a-4477-9a65-4a6095268a1b-kube-api-access-plxnf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.673323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqp4z\" (UniqueName: \"kubernetes.io/projected/e0616c07-ca03-4208-a925-cba77fe73ec1-kube-api-access-sqp4z\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.673422 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673329    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-host-slash\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.673422 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4xz\" (UniqueName: \"kubernetes.io/projected/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-kube-api-access-2r4xz\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.673657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-kubernetes\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.673728 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-socket-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.673728 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:52.673829 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-system-cni-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.673829 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.673785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.674242 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.674217    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:52.696726 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.696703    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhlqz"
Sep 17 17:33:52.705330 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.705302    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhlqz"
Sep 17 17:33:52.760740 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.760720    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:33:52.773991 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.773964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-iptables-alerter-script\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.774107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-tuned\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.774107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0616c07-ca03-4208-a925-cba77fe73ec1-host\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.774107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2024083-28b8-4e85-8dab-49e3b84bc155-konnectivity-ca\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.774107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qqj\" (UniqueName: \"kubernetes.io/projected/b8c30850-babf-4c71-8e02-ea922ec6732a-kube-api-access-69qqj\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-device-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whn2j\" (UniqueName: \"kubernetes.io/projected/56037b3a-b283-43e7-aad2-f0e19394874b-kube-api-access-whn2j\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-bin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-var-lib-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-sys-fs\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bnj\" (UniqueName: \"kubernetes.io/projected/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kube-api-access-b4bnj\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-systemd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.774315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrhb\" (UniqueName: \"kubernetes.io/projected/98409144-10a7-4d73-81f5-1575ffe6fc07-kube-api-access-7zrhb\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-var-lib-kubelet\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2024083-28b8-4e85-8dab-49e3b84bc155-agent-certs\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774444    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-cnibin\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-conf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774505    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:33:52.774652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-iptables-alerter-script\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.774995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774902    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-sys-fs\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.774995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-device-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.774995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-cnibin\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.774995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0616c07-ca03-4208-a925-cba77fe73ec1-host\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.775166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-var-lib-kubelet\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2024083-28b8-4e85-8dab-49e3b84bc155-konnectivity-ca\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.775166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-conf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.774510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-etc-kubernetes\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.775166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-netd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-config\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-kubelet-config\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysconfig\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-kubelet-config\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysconfig\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-tmp\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxnf\" (UniqueName: \"kubernetes.io/projected/de37802d-fd4a-4477-9a65-4a6095268a1b-kube-api-access-plxnf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.775411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-system-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775411    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-kubelet\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-host-slash\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4xz\" (UniqueName: \"kubernetes.io/projected/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-kube-api-access-2r4xz\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-cnibin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-kubelet\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775755    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-host-slash\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-etc-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-log-socket\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-slash\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.775871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-etc-selinux\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-bin\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.775995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-sysctl-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-env-overrides\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98409144-10a7-4d73-81f5-1575ffe6fc07-ovn-node-metrics-cert\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-etc-selinux\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjccl\" (UniqueName: \"kubernetes.io/projected/2aff5a10-19db-45d6-8463-2fdfea258bbf-kube-api-access-wjccl\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-run\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-registration-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-conf-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.776347    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-script-lib\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.776464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-run\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.776426    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:53.276389099 +0000 UTC m=+4.643520959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776457    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-registration-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56037b3a-b283-43e7-aad2-f0e19394874b-hosts-file\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56037b3a-b283-43e7-aad2-f0e19394874b-tmp-dir\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-dbus\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56037b3a-b283-43e7-aad2-f0e19394874b-hosts-file\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-systemd\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-host\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776713    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8caffdba-5c31-4028-8e2a-3574ca05bb23-dbus\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0616c07-ca03-4208-a925-cba77fe73ec1-serviceca\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqp4z\" (UniqueName: \"kubernetes.io/projected/e0616c07-ca03-4208-a925-cba77fe73ec1-kube-api-access-sqp4z\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-cni-binary-copy\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-multus-daemon-config\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56037b3a-b283-43e7-aad2-f0e19394874b-tmp-dir\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.776917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-k8s-cni-cncf-io\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777004    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-hostroot\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777028    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-systemd\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-multus-certs\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-host\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-netns\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-kubernetes\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-socket-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c30850-babf-4c71-8e02-ea922ec6732a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-kubernetes\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-multus\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-ovn\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777327    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0616c07-ca03-4208-a925-cba77fe73ec1-serviceca\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-system-cni-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.777994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-netns\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777424    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-node-log\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777429    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-socket-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c30850-babf-4c71-8e02-ea922ec6732a-system-cni-dir\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-modprobe-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-sys\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-socket-dir-parent\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777655    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-sys\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777671    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-modprobe-d\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6s89\" (UniqueName: \"kubernetes.io/projected/e96db70e-2d64-41ad-902d-531efbc42295-kube-api-access-w6s89\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-systemd-units\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-lib-modules\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de37802d-fd4a-4477-9a65-4a6095268a1b-lib-modules\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.778696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.777941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"5330a9d330e353403bdebed08bc7f829","Type":"ContainerStarted","Data":"03157de71b28e2a5480dbb6f0fd8df44b0b07de7c7ba9cf4ab67aa2bc2e74137"}
Sep 17 17:33:52.779420 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.778704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-etc-tuned\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.779420 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.779350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de37802d-fd4a-4477-9a65-4a6095268a1b-tmp\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.779662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.779636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" event={"ID":"afdef641cffa9a246bda35ec2b888902","Type":"ContainerStarted","Data":"e03d1506cd6be4874fbb6c501b033770641979039542e1d48f44f14618811329"}
Sep 17 17:33:52.780191 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.780167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2024083-28b8-4e85-8dab-49e3b84bc155-agent-certs\") pod \"konnectivity-agent-f5d72\" (UID: \"a2024083-28b8-4e85-8dab-49e3b84bc155\") " pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.784877 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.784843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qqj\" (UniqueName: \"kubernetes.io/projected/b8c30850-babf-4c71-8e02-ea922ec6732a-kube-api-access-69qqj\") pod \"multus-additional-cni-plugins-hgh7c\" (UID: \"b8c30850-babf-4c71-8e02-ea922ec6732a\") " pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.786326 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.786306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxnf\" (UniqueName: \"kubernetes.io/projected/de37802d-fd4a-4477-9a65-4a6095268a1b-kube-api-access-plxnf\") pod \"tuned-8lgck\" (UID: \"de37802d-fd4a-4477-9a65-4a6095268a1b\") " pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:52.786406 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.786375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4xz\" (UniqueName: \"kubernetes.io/projected/a00801ba-17a4-4fbc-aaa8-bbb5be9ed311-kube-api-access-2r4xz\") pod \"iptables-alerter-gz8xp\" (UID: \"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311\") " pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.786535 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.786466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whn2j\" (UniqueName: \"kubernetes.io/projected/56037b3a-b283-43e7-aad2-f0e19394874b-kube-api-access-whn2j\") pod \"node-resolver-xnbdh\" (UID: \"56037b3a-b283-43e7-aad2-f0e19394874b\") " pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:52.786594 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.786574    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bnj\" (UniqueName: \"kubernetes.io/projected/abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490-kube-api-access-b4bnj\") pod \"aws-ebs-csi-driver-node-94qx7\" (UID: \"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:52.788990 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.788968    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:52.788990 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.788993    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:52.789151 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.789007    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:52.789151 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.789060    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:53.289046884 +0000 UTC m=+4.656178743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:52.791076 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.791057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqp4z\" (UniqueName: \"kubernetes.io/projected/e0616c07-ca03-4208-a925-cba77fe73ec1-kube-api-access-sqp4z\") pod \"node-ca-c8n74\" (UID: \"e0616c07-ca03-4208-a925-cba77fe73ec1\") " pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.878746 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878746 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-cni-binary-copy\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-multus-daemon-config\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-k8s-cni-cncf-io\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-hostroot\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-multus-certs\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-netns\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-multus\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-hostroot\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-ovn\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878946    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-ovn\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.878952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878944    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-multus-certs\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-netns\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-netns\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-k8s-cni-cncf-io\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-node-log\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.878981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-multus\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879005    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-run-netns\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879033    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-node-log\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-socket-dir-parent\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6s89\" (UniqueName: \"kubernetes.io/projected/e96db70e-2d64-41ad-902d-531efbc42295-kube-api-access-w6s89\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-systemd-units\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.879128    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-socket-dir-parent\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-bin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879148    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-systemd-units\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-cni-bin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.879499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-var-lib-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:52.879192    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:33:53.379173801 +0000 UTC m=+4.746305674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-var-lib-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-systemd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-systemd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-run-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrhb\" (UniqueName: \"kubernetes.io/projected/98409144-10a7-4d73-81f5-1575ffe6fc07-kube-api-access-7zrhb\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-etc-kubernetes\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-netd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-etc-kubernetes\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-cni-binary-copy\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879462    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-netd\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e96db70e-2d64-41ad-902d-531efbc42295-multus-daemon-config\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-config\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.880301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-system-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-kubelet\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-cnibin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-system-cni-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879597    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-kubelet\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-kubelet\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-host-var-lib-kubelet\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-cnibin\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-etc-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-log-socket\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-etc-openvswitch\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-slash\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879709    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-log-socket\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-bin\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-slash\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-env-overrides\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-cni-bin\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98409144-10a7-4d73-81f5-1575ffe6fc07-ovn-node-metrics-cert\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjccl\" (UniqueName: \"kubernetes.io/projected/2aff5a10-19db-45d6-8463-2fdfea258bbf-kube-api-access-wjccl\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-conf-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-script-lib\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e96db70e-2d64-41ad-902d-531efbc42295-multus-conf-dir\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.879971    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98409144-10a7-4d73-81f5-1575ffe6fc07-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.880013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-config\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.880147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-env-overrides\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.881777 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.880459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98409144-10a7-4d73-81f5-1575ffe6fc07-ovnkube-script-lib\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.882213 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.882139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98409144-10a7-4d73-81f5-1575ffe6fc07-ovn-node-metrics-cert\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.891014 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.890958    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6s89\" (UniqueName: \"kubernetes.io/projected/e96db70e-2d64-41ad-902d-531efbc42295-kube-api-access-w6s89\") pod \"multus-89kdm\" (UID: \"e96db70e-2d64-41ad-902d-531efbc42295\") " pod="openshift-multus/multus-89kdm"
Sep 17 17:33:52.891426 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.891405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjccl\" (UniqueName: \"kubernetes.io/projected/2aff5a10-19db-45d6-8463-2fdfea258bbf-kube-api-access-wjccl\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:52.891559 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.891544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrhb\" (UniqueName: \"kubernetes.io/projected/98409144-10a7-4d73-81f5-1575ffe6fc07-kube-api-access-7zrhb\") pod \"ovnkube-node-zbx6h\" (UID: \"98409144-10a7-4d73-81f5-1575ffe6fc07\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:52.964600 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.964579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:33:52.976471 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.976455    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gz8xp"
Sep 17 17:33:52.983999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.983982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hgh7c"
Sep 17 17:33:52.989637 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.989618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c8n74"
Sep 17 17:33:52.996188 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:52.996170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8lgck"
Sep 17 17:33:53.002790 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.002772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7"
Sep 17 17:33:53.008169 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.008152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xnbdh"
Sep 17 17:33:53.014562 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.014545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89kdm"
Sep 17 17:33:53.020176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.020158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:33:53.282864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.282804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:53.282988 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.282900    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:53.282988 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.282949    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.282936382 +0000 UTC m=+5.650068240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:53.383361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.383332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:53.383502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.383375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:53.383502 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383493    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:53.383603 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383541    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.383529517 +0000 UTC m=+5.750661375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:53.383603 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383493    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:53.383603 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383591    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:53.383603 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383604    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:53.383769 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:53.383683    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.383668962 +0000 UTC m=+5.750800841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:53.547355 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.547327    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc04ec9_a1d5_4d8a_8f23_7a7fcc23b490.slice/crio-8f98d0008ca2114fbdf2ddf64654bf062a9fca173b74987a505f76f11016e68d WatchSource:0}: Error finding container 8f98d0008ca2114fbdf2ddf64654bf062a9fca173b74987a505f76f11016e68d: Status 404 returned error can't find the container with id 8f98d0008ca2114fbdf2ddf64654bf062a9fca173b74987a505f76f11016e68d
Sep 17 17:33:53.550279 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.550249    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00801ba_17a4_4fbc_aaa8_bbb5be9ed311.slice/crio-c5e229d7ab7b0ea13c2eb3c897779fe22fca2fc12d5fa015ead89a2887343b65 WatchSource:0}: Error finding container c5e229d7ab7b0ea13c2eb3c897779fe22fca2fc12d5fa015ead89a2887343b65: Status 404 returned error can't find the container with id c5e229d7ab7b0ea13c2eb3c897779fe22fca2fc12d5fa015ead89a2887343b65
Sep 17 17:33:53.551694 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.551669    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde37802d_fd4a_4477_9a65_4a6095268a1b.slice/crio-2f4f0f40f107f4f81502c57a95da619ac256fd249e2a3abf59ee428ba8a31178 WatchSource:0}: Error finding container 2f4f0f40f107f4f81502c57a95da619ac256fd249e2a3abf59ee428ba8a31178: Status 404 returned error can't find the container with id 2f4f0f40f107f4f81502c57a95da619ac256fd249e2a3abf59ee428ba8a31178
Sep 17 17:33:53.552764 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.552690    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56037b3a_b283_43e7_aad2_f0e19394874b.slice/crio-52660b744f6f1adf4f5c76b0e40f526e9f13ac68732d741421e9809c7ccfa216 WatchSource:0}: Error finding container 52660b744f6f1adf4f5c76b0e40f526e9f13ac68732d741421e9809c7ccfa216: Status 404 returned error can't find the container with id 52660b744f6f1adf4f5c76b0e40f526e9f13ac68732d741421e9809c7ccfa216
Sep 17 17:33:53.554317 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.554204    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2024083_28b8_4e85_8dab_49e3b84bc155.slice/crio-bc18c56418b9b2298d49513c2010ba3e4b337413610c57c4803373f5969f318a WatchSource:0}: Error finding container bc18c56418b9b2298d49513c2010ba3e4b337413610c57c4803373f5969f318a: Status 404 returned error can't find the container with id bc18c56418b9b2298d49513c2010ba3e4b337413610c57c4803373f5969f318a
Sep 17 17:33:53.555129 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.555081    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98409144_10a7_4d73_81f5_1575ffe6fc07.slice/crio-b98223080912310c82473426133e4b83c78a13efbd3762f071f05f7cf833ec12 WatchSource:0}: Error finding container b98223080912310c82473426133e4b83c78a13efbd3762f071f05f7cf833ec12: Status 404 returned error can't find the container with id b98223080912310c82473426133e4b83c78a13efbd3762f071f05f7cf833ec12
Sep 17 17:33:53.556712 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.556012    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c30850_babf_4c71_8e02_ea922ec6732a.slice/crio-8b763b39cdb6e986df8180da783ac7f6dbd1e44223673778f8acef7da1d1b3bb WatchSource:0}: Error finding container 8b763b39cdb6e986df8180da783ac7f6dbd1e44223673778f8acef7da1d1b3bb: Status 404 returned error can't find the container with id 8b763b39cdb6e986df8180da783ac7f6dbd1e44223673778f8acef7da1d1b3bb
Sep 17 17:33:53.557743 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.557033    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0616c07_ca03_4208_a925_cba77fe73ec1.slice/crio-8b78bb7d366bba0f8ec4ada0553074405ba34d5473d1fc78f9000421b39f4d00 WatchSource:0}: Error finding container 8b78bb7d366bba0f8ec4ada0553074405ba34d5473d1fc78f9000421b39f4d00: Status 404 returned error can't find the container with id 8b78bb7d366bba0f8ec4ada0553074405ba34d5473d1fc78f9000421b39f4d00
Sep 17 17:33:53.558213 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:33:53.557983    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96db70e_2d64_41ad_902d_531efbc42295.slice/crio-545a05444a85203965d6050f83db52d44885a08e42609a587025987e7e27963a WatchSource:0}: Error finding container 545a05444a85203965d6050f83db52d44885a08e42609a587025987e7e27963a: Status 404 returned error can't find the container with id 545a05444a85203965d6050f83db52d44885a08e42609a587025987e7e27963a
Sep 17 17:33:53.708197 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.708054    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:52 +0000 UTC" deadline="2028-02-16 22:20:25.264589419 +0000 UTC"
Sep 17 17:33:53.708197 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.708197    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12412h46m31.556398127s"
Sep 17 17:33:53.782165 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.782131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7" event={"ID":"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490","Type":"ContainerStarted","Data":"8f98d0008ca2114fbdf2ddf64654bf062a9fca173b74987a505f76f11016e68d"}
Sep 17 17:33:53.783337 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.783309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c8n74" event={"ID":"e0616c07-ca03-4208-a925-cba77fe73ec1","Type":"ContainerStarted","Data":"8b78bb7d366bba0f8ec4ada0553074405ba34d5473d1fc78f9000421b39f4d00"}
Sep 17 17:33:53.784486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.784463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"b98223080912310c82473426133e4b83c78a13efbd3762f071f05f7cf833ec12"}
Sep 17 17:33:53.785576 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.785551    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f5d72" event={"ID":"a2024083-28b8-4e85-8dab-49e3b84bc155","Type":"ContainerStarted","Data":"bc18c56418b9b2298d49513c2010ba3e4b337413610c57c4803373f5969f318a"}
Sep 17 17:33:53.786600 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.786580    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xnbdh" event={"ID":"56037b3a-b283-43e7-aad2-f0e19394874b","Type":"ContainerStarted","Data":"52660b744f6f1adf4f5c76b0e40f526e9f13ac68732d741421e9809c7ccfa216"}
Sep 17 17:33:53.787717 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.787693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gz8xp" event={"ID":"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311","Type":"ContainerStarted","Data":"c5e229d7ab7b0ea13c2eb3c897779fe22fca2fc12d5fa015ead89a2887343b65"}
Sep 17 17:33:53.790632 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.790566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" event={"ID":"afdef641cffa9a246bda35ec2b888902","Type":"ContainerStarted","Data":"a6bb119456c4c7f458a535a257c6874aef10b69f0c125a68d8d070c66b25df5f"}
Sep 17 17:33:53.792317 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.792297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"5330a9d330e353403bdebed08bc7f829","Type":"ContainerStarted","Data":"83ae53dce785afa4373dd58625265174017a0d3a87aa893b6f0609292ee4feda"}
Sep 17 17:33:53.793340 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.793315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89kdm" event={"ID":"e96db70e-2d64-41ad-902d-531efbc42295","Type":"ContainerStarted","Data":"545a05444a85203965d6050f83db52d44885a08e42609a587025987e7e27963a"}
Sep 17 17:33:53.794280 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.794263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerStarted","Data":"8b763b39cdb6e986df8180da783ac7f6dbd1e44223673778f8acef7da1d1b3bb"}
Sep 17 17:33:53.795245 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.795226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8lgck" event={"ID":"de37802d-fd4a-4477-9a65-4a6095268a1b","Type":"ContainerStarted","Data":"2f4f0f40f107f4f81502c57a95da619ac256fd249e2a3abf59ee428ba8a31178"}
Sep 17 17:33:53.830989 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:53.830941    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" podStartSLOduration=2.8309298050000002 podStartE2EDuration="2.830929805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:53.809609832 +0000 UTC m=+5.176741710" watchObservedRunningTime="2026-09-17 17:33:53.830929805 +0000 UTC m=+5.198061685"
Sep 17 17:33:54.291333 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.290724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:54.291333 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.290863    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.291333 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.290923    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:56.290905372 +0000 UTC m=+7.658037245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.392026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.392076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392248    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392301    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:33:56.392284052 +0000 UTC m=+7.759415913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392686    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392702    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392714    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.392789 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.392757    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:56.392742103 +0000 UTC m=+7.759873968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.709191 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.709026    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:52 +0000 UTC" deadline="2028-04-24 02:18:53.826746227 +0000 UTC"
Sep 17 17:33:54.709623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.709186    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14024h44m59.117564911s"
Sep 17 17:33:54.773290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.773265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:54.773421 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.773376    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:33:54.773780 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.773758    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:54.773888 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.773854    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:33:54.775469 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.775446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:54.775580 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:54.775552    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:33:54.821080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.816143    2575 generic.go:356] "Generic (PLEG): container finished" podID="5330a9d330e353403bdebed08bc7f829" containerID="83ae53dce785afa4373dd58625265174017a0d3a87aa893b6f0609292ee4feda" exitCode=0
Sep 17 17:33:54.821080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:54.816951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"5330a9d330e353403bdebed08bc7f829","Type":"ContainerDied","Data":"83ae53dce785afa4373dd58625265174017a0d3a87aa893b6f0609292ee4feda"}
Sep 17 17:33:55.839179 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:55.839088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"5330a9d330e353403bdebed08bc7f829","Type":"ContainerStarted","Data":"a27e1cf6f01443806f80f45452b6ed66f4d2aabdb51c3eb788a9b84a974c626f"}
Sep 17 17:33:56.306028 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.305950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:56.306190 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.306137    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:56.306250 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.306198    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:00.306179583 +0000 UTC m=+11.673311460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.406734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.406818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.406966    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.406989    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.407001    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.407054    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:00.407036349 +0000 UTC m=+11.774168222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.407490    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:56.407574 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.407539    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:00.407522398 +0000 UTC m=+11.774654270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.772581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.772581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.772758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.772833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:56.772887    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:56.773035 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:56.772963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:33:58.773136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:58.773091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:33:58.773590 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:58.773088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:33:58.773590 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:58.773233    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:33:58.773590 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:58.773335    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:33:58.773590 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:33:58.773391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:33:58.773590 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:33:58.773503    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:00.342942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.342396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:00.342942 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.342537    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:00.342942 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.342605    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.342585846 +0000 UTC m=+19.709717751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:00.443677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.443567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:00.443677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.443622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443745    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443758    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443833    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.443812034 +0000 UTC m=+19.810943898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443770    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443859    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:00.443911 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.443907    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.443894851 +0000 UTC m=+19.811026731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:00.774073 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.774002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:00.774230 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.774129    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:00.774471 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.774454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:00.774597 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.774579    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:00.774650 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:00.774629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:00.774726 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:00.774710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:02.772948 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:02.772878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:02.773422 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:02.772879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:02.773422 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:02.773007    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:02.773422 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:02.773008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:02.773422 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:02.773126    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:02.773422 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:02.773249    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:04.773088 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:04.773050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:04.773678 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:04.773175    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:04.773678 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:04.773468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:04.773678 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:04.773533    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:04.773859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:04.773676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:04.773859 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:04.773799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:06.773062 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:06.773020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:06.773409 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:06.773069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:06.773409 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:06.773192    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:06.773409 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:06.773225    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:06.773409 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:06.773267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:06.773409 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:06.773330    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:08.406675 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.406638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:08.407133 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.406745    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.407133 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.406799    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret podName:8caffdba-5c31-4028-8e2a-3574ca05bb23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:24.406784887 +0000 UTC m=+35.773916745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret") pod "global-pull-secret-syncer-fxgnf" (UID: "8caffdba-5c31-4028-8e2a-3574ca05bb23") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.507553 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.507519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:08.507553 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.507556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507660    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507676    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507693    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507706    2575 projected.go:196] Error preparing data for projected volume kube-api-access-7qvd8 for pod openshift-network-diagnostics/network-check-target-vzdrl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507719    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:24.507701216 +0000 UTC m=+35.874833097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.507771 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.507740    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8 podName:053196ed-d593-4a70-a5cf-99164b6aca3d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:24.507729738 +0000 UTC m=+35.874861610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qvd8" (UniqueName: "kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8") pod "network-check-target-vzdrl" (UID: "053196ed-d593-4a70-a5cf-99164b6aca3d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.773132 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.773050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:08.773132 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.773081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:08.773132 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:08.773062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:08.773392 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.773172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:08.773392 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.773257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:08.773392 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:08.773360    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:10.773815 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:10.773793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:10.774076 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:10.773891    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:10.774076 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:10.773980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:10.774076 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:10.774066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:10.774205 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:10.774087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:10.774205 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:10.774155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:11.872554 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.872173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89kdm" event={"ID":"e96db70e-2d64-41ad-902d-531efbc42295","Type":"ContainerStarted","Data":"b0fd9aa52011e814cadab6627633648fb92495e1e9a7652a4797387347dad40e"}
Sep 17 17:34:11.873776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.873754    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="838c9b7bb04079ad30c69ccaa8339b51b28ed6adaff6ca384a0414427dbd7d9e" exitCode=0
Sep 17 17:34:11.873889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.873831    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"838c9b7bb04079ad30c69ccaa8339b51b28ed6adaff6ca384a0414427dbd7d9e"}
Sep 17 17:34:11.875486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.875465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8lgck" event={"ID":"de37802d-fd4a-4477-9a65-4a6095268a1b","Type":"ContainerStarted","Data":"40a2ebc0022ffa18691acc32efbb56892897dc031b8639417e3de56176d12729"}
Sep 17 17:34:11.876854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.876835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7" event={"ID":"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490","Type":"ContainerStarted","Data":"28d1f24b0803774af791f1bfe869b1f1d428d10aeb033aad418ddcbd902107ad"}
Sep 17 17:34:11.878339 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.878315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c8n74" event={"ID":"e0616c07-ca03-4208-a925-cba77fe73ec1","Type":"ContainerStarted","Data":"43dd2c9fdbc3fb0a178096fdf42b1fe51ece6e6472635c0d778a9e5193b135f0"}
Sep 17 17:34:11.880957 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.880938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:34:11.881621 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881604    2575 generic.go:356] "Generic (PLEG): container finished" podID="98409144-10a7-4d73-81f5-1575ffe6fc07" containerID="26665662e5aec12d171002b61285bfcf47b822c7e717579b08177828cac08b3f" exitCode=1
Sep 17 17:34:11.881679 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"c58069db4c2740dc852a74d2b829a6a9eb246e61a82e93a5448714b526348e6d"}
Sep 17 17:34:11.881712 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"63b418d81fcef1bb32443d66dbcbb28b93cbe5e1a6ec63b1faf7522fe5ecd56e"}
Sep 17 17:34:11.881758 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"c0932aeea81a168a36f6cb3f1f9b200ba195ef0920d315cc78a58c8b46993bb5"}
Sep 17 17:34:11.881795 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"ed0f445cd29cd2db3f70bd5471d9596be3d16ad28ddaa6b7b7307ba56ddc49f4"}
Sep 17 17:34:11.881842 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerDied","Data":"26665662e5aec12d171002b61285bfcf47b822c7e717579b08177828cac08b3f"}
Sep 17 17:34:11.881842 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.881823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"ef47061653607b3c5b76442a155500cf444c3e1fd7bc0c3854e456e78be177e2"}
Sep 17 17:34:11.882897 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.882878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f5d72" event={"ID":"a2024083-28b8-4e85-8dab-49e3b84bc155","Type":"ContainerStarted","Data":"1005056a3d13d9275c58a3cf3c8a581918deea4aff8b819d808cc61d768db325"}
Sep 17 17:34:11.884131 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.884088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xnbdh" event={"ID":"56037b3a-b283-43e7-aad2-f0e19394874b","Type":"ContainerStarted","Data":"d89baafbb26431f625dfa80ff5020941628747092ee976c3d6cf591b6503e551"}
Sep 17 17:34:11.892834 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.892798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-89kdm" podStartSLOduration=3.615588309 podStartE2EDuration="20.892787649s" totalImagesPullingTime="17.27719934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.89035423 +0000 UTC m=+23.257486114" watchObservedRunningTime="2026-09-17 17:34:11.892787649 +0000 UTC m=+23.259919529"
Sep 17 17:34:11.893149 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.893128    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" podStartSLOduration=20.89312146 podStartE2EDuration="20.89312146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:55.856479943 +0000 UTC m=+7.223611827" watchObservedRunningTime="2026-09-17 17:34:11.89312146 +0000 UTC m=+23.260253340"
Sep 17 17:34:11.907150 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.907119    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c8n74" podStartSLOduration=3.671823378 podStartE2EDuration="20.907111643s" totalImagesPullingTime="17.235288265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.906858942 +0000 UTC m=+23.273990823" watchObservedRunningTime="2026-09-17 17:34:11.907111643 +0000 UTC m=+23.274243514"
Sep 17 17:34:11.922986 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.922954    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8lgck" podStartSLOduration=3.680386714 podStartE2EDuration="20.922944424s" totalImagesPullingTime="17.24255771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.922854248 +0000 UTC m=+23.289986128" watchObservedRunningTime="2026-09-17 17:34:11.922944424 +0000 UTC m=+23.290076305"
Sep 17 17:34:11.939342 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.939312    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f5d72" podStartSLOduration=8.965700419000001 podStartE2EDuration="21.939301406s" totalImagesPullingTime="12.973600987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.938956348 +0000 UTC m=+23.306088232" watchObservedRunningTime="2026-09-17 17:34:11.939301406 +0000 UTC m=+23.306433286"
Sep 17 17:34:11.987134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:11.987085    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xnbdh" podStartSLOduration=3.746983053 podStartE2EDuration="20.987075169s" totalImagesPullingTime="17.240092116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.986790603 +0000 UTC m=+23.353922484" watchObservedRunningTime="2026-09-17 17:34:11.987075169 +0000 UTC m=+23.354207051"
Sep 17 17:34:12.300436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.300409    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:12.739332 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.739242    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:12.300433383Z","UUID":"2c82700f-88b2-403a-8365-2ff70d47029a","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:12.740754 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.740735    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:12.740754 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.740761    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:12.772703 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.772681    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:12.772823 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.772683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:12.772823 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:12.772808    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:12.772930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.772818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:12.772930 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:12.772905    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:12.773028 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:12.772983    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:12.888992 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.888720    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gz8xp" event={"ID":"a00801ba-17a4-4fbc-aaa8-bbb5be9ed311","Type":"ContainerStarted","Data":"75f1b5928c91fc3afd2024240ca840f53c1801c4ee3db4fe071c1f557b5d6d01"}
Sep 17 17:34:12.891563 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.891052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7" event={"ID":"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490","Type":"ContainerStarted","Data":"2c041bf91976e1dfd36822855ff26a79b2d1d909ae6f0bfdb5fd675257290b82"}
Sep 17 17:34:12.904005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:12.903957    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gz8xp" podStartSLOduration=5.692171427 podStartE2EDuration="22.903943728s" totalImagesPullingTime="17.211772301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:12.903465848 +0000 UTC m=+24.270597730" watchObservedRunningTime="2026-09-17 17:34:12.903943728 +0000 UTC m=+24.271075608"
Sep 17 17:34:13.155639 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.155611    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:34:13.156182 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.156163    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:34:13.596548 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.596518    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:34:13.597072 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.597050    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f5d72"
Sep 17 17:34:13.897978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.897929    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:34:13.898460 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.898430    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"c206217b104b7ed4974ef1933de8136d3df90036af6ce5660836b376639fc20a"}
Sep 17 17:34:13.901061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.901038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7" event={"ID":"abc04ec9-a1d5-4d8a-8f23-7a7fcc23b490","Type":"ContainerStarted","Data":"61c4f9f332490a7d68977c2f8ed00bb512617788abd1dcfa416e69def366342c"}
Sep 17 17:34:13.920784 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:13.920745    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94qx7" podStartSLOduration=3.66210307 podStartE2EDuration="22.920733166s" totalImagesPullingTime="19.258630096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:13.920547748 +0000 UTC m=+25.287679631" watchObservedRunningTime="2026-09-17 17:34:13.920733166 +0000 UTC m=+25.287865043"
Sep 17 17:34:14.772947 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:14.772882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:14.773147 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:14.772998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:14.773147 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:14.773008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:14.773147 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:14.773005    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:14.773147 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:14.773125    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:14.773351 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:14.773204    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:16.772444 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.772224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:16.773370 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.772233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:16.773370 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:16.772545    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:16.773370 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.772256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:16.773370 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:16.772645    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:16.773370 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:16.772714    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:16.909283 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.909254    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="ce7c889f39bc282b2701c8d5dc7e5828055ce1b8836df832065bb88adbaa7f9a" exitCode=0
Sep 17 17:34:16.909459 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.909344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"ce7c889f39bc282b2701c8d5dc7e5828055ce1b8836df832065bb88adbaa7f9a"}
Sep 17 17:34:16.913466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.913444    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:34:16.913896 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.913868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"8bc9976c6e221cc292852c73aaef2e168a0bab25bedd2f75ee9ee5ba386280f9"}
Sep 17 17:34:16.914145 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.914127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:16.914340 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.914323    2575 scope.go:117] "RemoveContainer" containerID="26665662e5aec12d171002b61285bfcf47b822c7e717579b08177828cac08b3f"
Sep 17 17:34:16.931368 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:16.931344    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:17.919970 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.919808    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:34:17.920463 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.920345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" event={"ID":"98409144-10a7-4d73-81f5-1575ffe6fc07","Type":"ContainerStarted","Data":"d869ee26f2c25279e105d23711561ebd05e6aa7128698ee130b44f28ec32e53d"}
Sep 17 17:34:17.920463 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.920444    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:17.920686 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.920666    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:17.934261 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.934231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:17.958248 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.958204    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" podStartSLOduration=9.710888242 podStartE2EDuration="26.958191809s" totalImagesPullingTime="17.247303567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:17.957855206 +0000 UTC m=+29.324987086" watchObservedRunningTime="2026-09-17 17:34:17.958191809 +0000 UTC m=+29.325323690"
Sep 17 17:34:17.993964 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.993944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxgnf"]
Sep 17 17:34:17.994085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:17.994072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:17.994194 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:17.994172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:18.000706 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.000674    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twm78"]
Sep 17 17:34:18.000797 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.000763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:18.000861 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:18.000838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:18.001396 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.001355    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzdrl"]
Sep 17 17:34:18.001530 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.001449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:18.001717 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:18.001530    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:18.925498 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.925460    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="ae60d7722070e5abf5ab0c5814ebe7692f42279974f99c1d92fd6de939b9cec1" exitCode=0
Sep 17 17:34:18.925942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.925553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"ae60d7722070e5abf5ab0c5814ebe7692f42279974f99c1d92fd6de939b9cec1"}
Sep 17 17:34:18.925942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:18.925616    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:19.772766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:19.772737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:19.772766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:19.772764    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:19.772955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:19.772863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:19.772955 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:19.772871    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:19.772955 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:19.772938    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:19.773051 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:19.772987    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:19.928504 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:19.928474    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:20.932749 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:20.932680    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="5157ee15f58e52fc2f996426aaf8fb36e2e457fa89c0afa8bf4a85beed4e09bd" exitCode=0
Sep 17 17:34:20.932749 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:20.932728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"5157ee15f58e52fc2f996426aaf8fb36e2e457fa89c0afa8bf4a85beed4e09bd"}
Sep 17 17:34:21.772613 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.772583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:21.772613 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.772618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:21.772837 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.772583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:21.772837 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:21.772696    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm78" podUID="2aff5a10-19db-45d6-8463-2fdfea258bbf"
Sep 17 17:34:21.772837 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:21.772781    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzdrl" podUID="053196ed-d593-4a70-a5cf-99164b6aca3d"
Sep 17 17:34:21.772959 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:21.772864    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxgnf" podUID="8caffdba-5c31-4028-8e2a-3574ca05bb23"
Sep 17 17:34:21.904962 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.904924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:21.905199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.905184    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:21.928498 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.928465    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" podUID="98409144-10a7-4d73-81f5-1575ffe6fc07" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:34:21.938107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:21.938058    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h" podUID="98409144-10a7-4d73-81f5-1575ffe6fc07" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:34:23.771119 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.770958    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeReady"
Sep 17 17:34:23.773385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.771698    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:23.773852 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.773829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:23.773971 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.773831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:23.774065 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.773831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:23.778266 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:23.778385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778057    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crvff\""
Sep 17 17:34:23.778385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778080    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qcgkp\""
Sep 17 17:34:23.778385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778085    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:23.778545 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778116    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:23.778545 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.778131    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:23.810890 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.810830    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:34:23.827361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.827331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.829372 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.829349    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:34:23.830090 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.830069    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:34:23.830999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.830979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jxpgb\""
Sep 17 17:34:23.831122 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.831012    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rbmz9"]
Sep 17 17:34:23.831122 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.831116    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:34:23.831407 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.831378    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:34:23.837048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.837028    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:34:23.841945 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.841875    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-88sj8"]
Sep 17 17:34:23.842041 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.841950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:23.844366 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.844345    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:23.844453 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.844424    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hrfbx\""
Sep 17 17:34:23.844453 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.844439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:23.844567 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.844552    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:23.854841 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.854823    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbmz9"]
Sep 17 17:34:23.854926 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.854851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-88sj8"]
Sep 17 17:34:23.854926 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.854920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:23.857172 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.857152    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:23.857172 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.857163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bd5nq\""
Sep 17 17:34:23.857323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.857234    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:23.932486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj76p\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:23.932766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:23.932752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.009789 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.009734    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.033707 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jl4\" (UniqueName: \"kubernetes.io/projected/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-kube-api-access-n9jl4\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.033822 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1773536c-31a2-436a-9f79-c5441b182f23-tmp-dir\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.033822 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.033951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.033951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.033951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.033951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.033986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-cert\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.034000    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1773536c-31a2-436a-9f79-c5441b182f23-config-volume\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.034015    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnldh\" (UniqueName: \"kubernetes.io/projected/1773536c-31a2-436a-9f79-c5441b182f23-kube-api-access-qnldh\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.034166 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.034092    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:24.534075077 +0000 UTC m=+35.901206951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:24.034461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj76p\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.034705 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034705 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.034927 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.034901    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.038630 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.038607    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.038630 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.038618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.044171 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.044152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.044272 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.044158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj76p\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.116984 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.116956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.134931 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.134907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-cert\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.134931 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.134932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1773536c-31a2-436a-9f79-c5441b182f23-config-volume\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.135134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.134950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnldh\" (UniqueName: \"kubernetes.io/projected/1773536c-31a2-436a-9f79-c5441b182f23-kube-api-access-qnldh\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.135134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.134983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.135134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.135019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jl4\" (UniqueName: \"kubernetes.io/projected/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-kube-api-access-n9jl4\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.135134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.135046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1773536c-31a2-436a-9f79-c5441b182f23-tmp-dir\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.135282 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.135168    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:24.135282 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.135240    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:24.635223935 +0000 UTC m=+36.002355799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:24.135394 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.135371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1773536c-31a2-436a-9f79-c5441b182f23-tmp-dir\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.135616 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.135597    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1773536c-31a2-436a-9f79-c5441b182f23-config-volume\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.137900 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.137881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-cert\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.146511 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.146481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnldh\" (UniqueName: \"kubernetes.io/projected/1773536c-31a2-436a-9f79-c5441b182f23-kube-api-access-qnldh\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.147497 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.147470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jl4\" (UniqueName: \"kubernetes.io/projected/282abe8e-7335-47ae-8ed9-ce6e2f1a36e5-kube-api-access-n9jl4\") pod \"ingress-canary-rbmz9\" (UID: \"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5\") " pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.153167 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.153150    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbmz9"
Sep 17 17:34:24.438122 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.438082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:24.440937 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.440911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8caffdba-5c31-4028-8e2a-3574ca05bb23-original-pull-secret\") pod \"global-pull-secret-syncer-fxgnf\" (UID: \"8caffdba-5c31-4028-8e2a-3574ca05bb23\") " pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:24.539476 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.539440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:24.539607 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.539489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:24.539676 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.539600    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:24.539676 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.539619    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:24.539676 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.539620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:24.539822 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.539684    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:25.539665545 +0000 UTC m=+36.906797419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:24.539822 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.539735    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:24.539822 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.539792    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.539778954 +0000 UTC m=+67.906910819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : secret "metrics-daemon-secret" not found
Sep 17 17:34:24.542404 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.542379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvd8\" (UniqueName: \"kubernetes.io/projected/053196ed-d593-4a70-a5cf-99164b6aca3d-kube-api-access-7qvd8\") pod \"network-check-target-vzdrl\" (UID: \"053196ed-d593-4a70-a5cf-99164b6aca3d\") " pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:24.640031 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.640004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:24.640211 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.640137    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:24.640211 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:24.640185    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:25.640172777 +0000 UTC m=+37.007304636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:24.686999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.686955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:24.700291 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:24.700237    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxgnf"
Sep 17 17:34:25.307149 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.307077    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxgnf"]
Sep 17 17:34:25.310253 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.310229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzdrl"]
Sep 17 17:34:25.310965 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.310942    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbmz9"]
Sep 17 17:34:25.334427 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:34:25.334404    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caffdba_5c31_4028_8e2a_3574ca05bb23.slice/crio-33481b204e7dd1ef9b0579c992d39a2ec3a438fc5302a3bb5556cd5bd5478e32 WatchSource:0}: Error finding container 33481b204e7dd1ef9b0579c992d39a2ec3a438fc5302a3bb5556cd5bd5478e32: Status 404 returned error can't find the container with id 33481b204e7dd1ef9b0579c992d39a2ec3a438fc5302a3bb5556cd5bd5478e32
Sep 17 17:34:25.335144 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:34:25.335116    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053196ed_d593_4a70_a5cf_99164b6aca3d.slice/crio-07013de27b8ee9736f3bd671f694a9bd4be0b94061ea65266cc787eb198ea33b WatchSource:0}: Error finding container 07013de27b8ee9736f3bd671f694a9bd4be0b94061ea65266cc787eb198ea33b: Status 404 returned error can't find the container with id 07013de27b8ee9736f3bd671f694a9bd4be0b94061ea65266cc787eb198ea33b
Sep 17 17:34:25.336074 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:34:25.336011    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282abe8e_7335_47ae_8ed9_ce6e2f1a36e5.slice/crio-a2fb93db2206afeed679f257021f9ea41c096f06660d6f2eba89978e928281b6 WatchSource:0}: Error finding container a2fb93db2206afeed679f257021f9ea41c096f06660d6f2eba89978e928281b6: Status 404 returned error can't find the container with id a2fb93db2206afeed679f257021f9ea41c096f06660d6f2eba89978e928281b6
Sep 17 17:34:25.430907 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.430874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:25.537207 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.537054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:25.547666 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.547643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:25.547845 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:25.547825    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:25.547896 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:25.547849    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:25.547933 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:25.547912    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.547890939 +0000 UTC m=+38.915022802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:25.648704 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.648678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:25.648783 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:25.648775    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:25.648827 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:25.648823    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.648810797 +0000 UTC m=+39.015942655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:25.952700 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.952507    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="a659c4eb74dcd7fef789a98187c4bb0ad97b6301d77a1e43116d5246a34f3017" exitCode=0
Sep 17 17:34:25.952700 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.952608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"a659c4eb74dcd7fef789a98187c4bb0ad97b6301d77a1e43116d5246a34f3017"}
Sep 17 17:34:25.954164 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.954137    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzdrl" event={"ID":"053196ed-d593-4a70-a5cf-99164b6aca3d","Type":"ContainerStarted","Data":"07013de27b8ee9736f3bd671f694a9bd4be0b94061ea65266cc787eb198ea33b"}
Sep 17 17:34:25.956935 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.956904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbmz9" event={"ID":"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5","Type":"ContainerStarted","Data":"a2fb93db2206afeed679f257021f9ea41c096f06660d6f2eba89978e928281b6"}
Sep 17 17:34:25.960499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:25.960468    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxgnf" event={"ID":"8caffdba-5c31-4028-8e2a-3574ca05bb23","Type":"ContainerStarted","Data":"33481b204e7dd1ef9b0579c992d39a2ec3a438fc5302a3bb5556cd5bd5478e32"}
Sep 17 17:34:26.969119 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:26.968377    2575 generic.go:356] "Generic (PLEG): container finished" podID="b8c30850-babf-4c71-8e02-ea922ec6732a" containerID="699001c5114177f213a93f68eaeab5244333bfabd5e212064eca3939d08040ee" exitCode=0
Sep 17 17:34:26.969119 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:26.968459    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerDied","Data":"699001c5114177f213a93f68eaeab5244333bfabd5e212064eca3939d08040ee"}
Sep 17 17:34:27.341466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:27.341381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.436502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:27.436439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.561064 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:27.561029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:27.561247 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:27.561203    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:27.561247 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:27.561220    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:27.561358 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:27.561283    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:31.561265449 +0000 UTC m=+42.928397334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:27.661600 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:27.661515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:27.661752 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:27.661700    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:27.661824 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:27.661768    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:31.661748714 +0000 UTC m=+43.028880573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:30.979615 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.979573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbmz9" event={"ID":"282abe8e-7335-47ae-8ed9-ce6e2f1a36e5","Type":"ContainerStarted","Data":"081f333bc7a19842ae1f310f57ea718980d23b7d7a48dded551cd8cb7815805e"}
Sep 17 17:34:30.981409 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.981372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxgnf" event={"ID":"8caffdba-5c31-4028-8e2a-3574ca05bb23","Type":"ContainerStarted","Data":"17b3ac4653a2fd593fac6578961254ca48d0e0b05756c0872401ff88deca479b"}
Sep 17 17:34:30.985624 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.985597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" event={"ID":"b8c30850-babf-4c71-8e02-ea922ec6732a","Type":"ContainerStarted","Data":"65f4f48d98c50d0e8d4c619c63fb3c11f6545b1e84e275bdceef0cab15551cae"}
Sep 17 17:34:30.987304 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.987280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzdrl" event={"ID":"053196ed-d593-4a70-a5cf-99164b6aca3d","Type":"ContainerStarted","Data":"a4d53852404624a159f5dc8bdcd14b95c52b89011675446cbe959ec225aff65e"}
Sep 17 17:34:30.987427 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.987406    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:34:30.997073 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:30.997023    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rbmz9" podStartSLOduration=3.000548844 podStartE2EDuration="7.997008347s" totalImagesPullingTime="4.996459503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:30.996155972 +0000 UTC m=+42.363287853" watchObservedRunningTime="2026-09-17 17:34:30.997008347 +0000 UTC m=+42.364140227"
Sep 17 17:34:31.015650 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:31.015605    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fxgnf" podStartSLOduration=35.007340821 podStartE2EDuration="40.015592474s" totalImagesPullingTime="5.008251653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:31.01530798 +0000 UTC m=+42.382439863" watchObservedRunningTime="2026-09-17 17:34:31.015592474 +0000 UTC m=+42.382724355"
Sep 17 17:34:31.042719 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:31.042684    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hgh7c" podStartSLOduration=13.332493949 podStartE2EDuration="41.042670743s" totalImagesPullingTime="27.710176794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:31.041515837 +0000 UTC m=+42.408647718" watchObservedRunningTime="2026-09-17 17:34:31.042670743 +0000 UTC m=+42.409802620"
Sep 17 17:34:31.058542 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:31.058503    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzdrl" podStartSLOduration=36.048458204 podStartE2EDuration="41.058491128s" totalImagesPullingTime="5.010032924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:31.058081348 +0000 UTC m=+42.425213229" watchObservedRunningTime="2026-09-17 17:34:31.058491128 +0000 UTC m=+42.425623022"
Sep 17 17:34:31.589486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:31.589447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:31.589616 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:31.589576    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:31.589616 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:31.589588    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:31.589704 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:31.589647    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.589635005 +0000 UTC m=+50.956766877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:31.689844 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:31.689817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:31.689947 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:31.689908    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:31.689988 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:31.689951    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.689939955 +0000 UTC m=+51.057071815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.642468 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:39.642428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:39.642888 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:39.642547    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:39.642888 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:39.642560    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:39.642888 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:39.642611    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.642597781 +0000 UTC m=+67.009729640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:39.743464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:39.743435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:39.743604 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:39.743540    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.743604 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:39.743585    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.743573013 +0000 UTC m=+67.110704871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:45.795254 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.795191    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66b4d4d798-wssc6"]
Sep 17 17:34:45.827802 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.827775    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66b4d4d798-wssc6"]
Sep 17 17:34:45.827927 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.827858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.830651 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.830626    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:34:45.830950 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.830929    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:34:45.831033 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.830943    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:34:45.831241 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.831222    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:34:45.831328 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.831235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q9s7x\""
Sep 17 17:34:45.831328 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.831228    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:34:45.831328 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.831231    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:34:45.881765 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.881739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-default-certificate\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.881869 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.881775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54stw\" (UniqueName: \"kubernetes.io/projected/67a217b5-fc9b-4eb4-ac95-d68b96579ded-kube-api-access-54stw\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.881869 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.881793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.881869 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.881827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.882023 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.881903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-stats-auth\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.982911 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.982887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54stw\" (UniqueName: \"kubernetes.io/projected/67a217b5-fc9b-4eb4-ac95-d68b96579ded-kube-api-access-54stw\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.983017 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.982915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.983017 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.982948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.983017 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.983007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-stats-auth\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.983202 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:45.983050    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.483034827 +0000 UTC m=+57.850166689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:45.983202 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.983116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-default-certificate\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.983202 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:45.983127    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:45.983202 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:45.983179    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.483162794 +0000 UTC m=+57.850294656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:34:45.986183 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.986158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-stats-auth\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.986268 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.986192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-default-certificate\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:45.996705 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:45.996686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54stw\" (UniqueName: \"kubernetes.io/projected/67a217b5-fc9b-4eb4-ac95-d68b96579ded-kube-api-access-54stw\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:46.485938 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:46.485879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:46.486038 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:46.485942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:46.486038 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:46.486010    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:46.486125 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:46.486038    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.486022323 +0000 UTC m=+58.853154186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:46.486125 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:46.486059    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.486052136 +0000 UTC m=+58.853183994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:34:47.494719 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:47.494681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:47.495176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:47.494778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:47.495176 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:47.494845    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:47.495176 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:47.494925    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.494908051 +0000 UTC m=+60.862039909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:34:47.495176 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:47.494939    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.494933011 +0000 UTC m=+60.862064871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:49.508222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:49.508182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:49.508595 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:49.508274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:49.508595 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:49.508331    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:49.508595 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:49.508390    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.508376261 +0000 UTC m=+64.875508120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:49.508595 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:49.508406    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.508400022 +0000 UTC m=+64.875531881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:34:51.938884 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:51.938857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbx6h"
Sep 17 17:34:52.803392 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.803355    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r6jkw"]
Sep 17 17:34:52.834658 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.834635    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r6jkw"]
Sep 17 17:34:52.834791 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.834728    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:52.837088 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.837050    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:34:52.837088 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.837050    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.837378 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.837357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:34:52.837518 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.837499    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7fflr\""
Sep 17 17:34:52.837737 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.837723    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.932039 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.932016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfvk\" (UniqueName: \"kubernetes.io/projected/841de304-78b6-4f84-8b77-ff547166887d-kube-api-access-hbfvk\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:52.932163 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.932051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:52.932206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.932162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/841de304-78b6-4f84-8b77-ff547166887d-crio-socket\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:52.932206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.932180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:52.932268 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:52.932205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/841de304-78b6-4f84-8b77-ff547166887d-data-volume\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032605 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfvk\" (UniqueName: \"kubernetes.io/projected/841de304-78b6-4f84-8b77-ff547166887d-kube-api-access-hbfvk\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/841de304-78b6-4f84-8b77-ff547166887d-crio-socket\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/841de304-78b6-4f84-8b77-ff547166887d-data-volume\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/841de304-78b6-4f84-8b77-ff547166887d-crio-socket\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.032819    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.032829    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.032901    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.532881834 +0000 UTC m=+64.900013712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.032925 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.032924    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.532915308 +0000 UTC m=+64.900047168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.033265 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.032990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/841de304-78b6-4f84-8b77-ff547166887d-data-volume\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.041429 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.041408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfvk\" (UniqueName: \"kubernetes.io/projected/841de304-78b6-4f84-8b77-ff547166887d-kube-api-access-hbfvk\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.535735 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.535706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.535842 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.535742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:53.535842 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.535775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:34:53.535842 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:53.535802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535840    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535862    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:35:01.53584721 +0000 UTC m=+72.902979069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535888    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535912    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.535893226 +0000 UTC m=+65.903025124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535928    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:34:53.535960 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535934    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.535923319 +0000 UTC m=+65.903055183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.536191 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:53.535981    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:35:01.535969312 +0000 UTC m=+72.903101210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:34:54.430624 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.430593    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"]
Sep 17 17:34:54.432855 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.432834    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.435554 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.435532    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:34:54.436464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436440    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:34:54.436564 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436484    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:34:54.436564 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436484    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:34:54.436669 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436485    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:34:54.436720 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436691    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:34:54.436858 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.436842    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:34:54.445817 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.445791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"]
Sep 17 17:34:54.543129 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk6d\" (UniqueName: \"kubernetes.io/projected/bf159bf2-d28f-4574-a08b-b2b389793fa0-kube-api-access-5lk6d\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.543231 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf159bf2-d28f-4574-a08b-b2b389793fa0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.543231 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:54.543231 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:54.543231 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.543381 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.543381 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.543381 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:54.543261    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:54.543381 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:54.543323    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.543304873 +0000 UTC m=+67.910436732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:54.543533 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:54.543387    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:54.543533 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:54.543436    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.543422658 +0000 UTC m=+67.910554537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:54.543533 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.543495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644395 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf159bf2-d28f-4574-a08b-b2b389793fa0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644491 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644491 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644491 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644612 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.644612 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.644605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk6d\" (UniqueName: \"kubernetes.io/projected/bf159bf2-d28f-4574-a08b-b2b389793fa0-kube-api-access-5lk6d\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.645260 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.645231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf159bf2-d28f-4574-a08b-b2b389793fa0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.647422 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.647389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.647541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.647452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-ca\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.647541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.647498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.647655 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.647595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf159bf2-d28f-4574-a08b-b2b389793fa0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.653565 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.653543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk6d\" (UniqueName: \"kubernetes.io/projected/bf159bf2-d28f-4574-a08b-b2b389793fa0-kube-api-access-5lk6d\") pod \"cluster-proxy-proxy-agent-588f99bb8d-qmfxx\" (UID: \"bf159bf2-d28f-4574-a08b-b2b389793fa0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:54.748329 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:54.748268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:34:55.023953 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:55.023888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"]
Sep 17 17:34:55.026880 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:34:55.026849    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf159bf2_d28f_4574_a08b_b2b389793fa0.slice/crio-0fad7466d3ee7568caf66592be4d9fe6b0fc2a375848df5b78161f78c7b56f86 WatchSource:0}: Error finding container 0fad7466d3ee7568caf66592be4d9fe6b0fc2a375848df5b78161f78c7b56f86: Status 404 returned error can't find the container with id 0fad7466d3ee7568caf66592be4d9fe6b0fc2a375848df5b78161f78c7b56f86
Sep 17 17:34:55.048694 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:55.048666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerStarted","Data":"0fad7466d3ee7568caf66592be4d9fe6b0fc2a375848df5b78161f78c7b56f86"}
Sep 17 17:34:55.651743 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:55.651701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:34:55.652223 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:55.651903    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:55.652223 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:55.651927    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f5f859bc4-kz5gr: secret "image-registry-tls" not found
Sep 17 17:34:55.652223 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:55.652002    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls podName:0a9268b1-4cea-4e84-8295-0d0df76b4df4 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:27.651978462 +0000 UTC m=+99.019110358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls") pod "image-registry-7f5f859bc4-kz5gr" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4") : secret "image-registry-tls" not found
Sep 17 17:34:55.752296 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:55.752269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:34:55.752435 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:55.752417    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:55.752482 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:55.752472    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls podName:1773536c-31a2-436a-9f79-c5441b182f23 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:27.752458933 +0000 UTC m=+99.119590837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls") pod "dns-default-88sj8" (UID: "1773536c-31a2-436a-9f79-c5441b182f23") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:55.767845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:55.767822    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xnbdh_56037b3a-b283-43e7-aad2-f0e19394874b/dns-node-resolver/0.log"
Sep 17 17:34:56.366832 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:56.366809    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c8n74_e0616c07-ca03-4208-a925-cba77fe73ec1/node-ca/0.log"
Sep 17 17:34:56.559364 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:56.559334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:56.559517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:56.559375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:34:56.559517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:56.559462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:34:56.559517 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559474    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:56.559671 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559539    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.559519431 +0000 UTC m=+71.926651307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:56.559671 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559602    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:56.559671 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559652    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs podName:2aff5a10-19db-45d6-8463-2fdfea258bbf nodeName:}" failed. No retries permitted until 2026-09-17 17:36:00.5596365 +0000 UTC m=+131.926768390 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs") pod "network-metrics-daemon-twm78" (UID: "2aff5a10-19db-45d6-8463-2fdfea258bbf") : secret "metrics-daemon-secret" not found
Sep 17 17:34:56.559841 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559740    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:56.559841 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:34:56.559777    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.559765059 +0000 UTC m=+71.926896922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:57.565989 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:34:57.565928    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbmz9_282abe8e-7335-47ae-8ed9-ce6e2f1a36e5/serve-healthcheck-canary/0.log"
Sep 17 17:35:00.066986 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.066944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerStarted","Data":"1a2bbe00f51d2a477b8a173cfc14a4b7d94c597dc23cc5f5b1bca003458d3b6e"}
Sep 17 17:35:00.067407 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.066998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerStarted","Data":"ac3a57952120f0f926554f37aff5fe2585abad9b4a27276489480d718262d678"}
Sep 17 17:35:00.067407 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.067020    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerStarted","Data":"8fe6e4b6a3a37e3627c82b268bfcb517066dcc90358fbfca59c90dc8d0096d98"}
Sep 17 17:35:00.088217 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.088169    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" podStartSLOduration=1.9788287100000002 podStartE2EDuration="6.088153923s" totalImagesPullingTime="4.109325213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:00.087856989 +0000 UTC m=+71.454988871" watchObservedRunningTime="2026-09-17 17:35:00.088153923 +0000 UTC m=+71.455285798"
Sep 17 17:35:00.591820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.591780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:00.591820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:00.591821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:00.592046 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:00.591903    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:35:00.592046 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:00.591970    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:00.592046 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:00.591975    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.591960066 +0000 UTC m=+79.959091925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : configmap "kube-rbac-proxy" not found
Sep 17 17:35:00.592046 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:00.592022    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.59201035 +0000 UTC m=+79.959142208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:01.598856 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:01.598807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:01.598856 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:01.598866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:01.599426 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:01.598934    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:35:01.599426 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:01.598996    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.598980693 +0000 UTC m=+88.966112552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : secret "router-metrics-certs-default" not found
Sep 17 17:35:01.599426 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:01.599010    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle podName:67a217b5-fc9b-4eb4-ac95-d68b96579ded nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.599002713 +0000 UTC m=+88.966134572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle") pod "router-default-66b4d4d798-wssc6" (UID: "67a217b5-fc9b-4eb4-ac95-d68b96579ded") : configmap references non-existent config key: service-ca.crt
Sep 17 17:35:01.994632 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:01.994569    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzdrl"
Sep 17 17:35:04.751183 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:04.751139    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" podUID="bf159bf2-d28f-4574-a08b-b2b389793fa0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:35:08.652715 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:08.652677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:08.652715 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:08.652715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:08.653319 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:08.652828    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:08.653319 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:08.652888    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls podName:841de304-78b6-4f84-8b77-ff547166887d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.652872921 +0000 UTC m=+96.020004779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6jkw" (UID: "841de304-78b6-4f84-8b77-ff547166887d") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:08.653319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:08.653223    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/841de304-78b6-4f84-8b77-ff547166887d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:14.749595 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:14.749542    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" podUID="bf159bf2-d28f-4574-a08b-b2b389793fa0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:35:17.615587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.615542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:17.615587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.615597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:17.616239 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.616213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a217b5-fc9b-4eb4-ac95-d68b96579ded-service-ca-bundle\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:17.618020 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.617997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a217b5-fc9b-4eb4-ac95-d68b96579ded-metrics-certs\") pod \"router-default-66b4d4d798-wssc6\" (UID: \"67a217b5-fc9b-4eb4-ac95-d68b96579ded\") " pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:17.640365 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.640333    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q9s7x\""
Sep 17 17:35:17.648600 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.648577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:17.929595 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:17.929528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66b4d4d798-wssc6"]
Sep 17 17:35:17.932428 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:17.932398    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a217b5_fc9b_4eb4_ac95_d68b96579ded.slice/crio-bf8d1f97dc98e2b8af3a011488811b253523fdcd13812a9309524f74e8316eff WatchSource:0}: Error finding container bf8d1f97dc98e2b8af3a011488811b253523fdcd13812a9309524f74e8316eff: Status 404 returned error can't find the container with id bf8d1f97dc98e2b8af3a011488811b253523fdcd13812a9309524f74e8316eff
Sep 17 17:35:18.121644 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:18.121613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66b4d4d798-wssc6" event={"ID":"67a217b5-fc9b-4eb4-ac95-d68b96579ded","Type":"ContainerStarted","Data":"76d3066d85f3e5acee98cd648436cf4a00b67023f6982c90b24a8ad64efd83ce"}
Sep 17 17:35:18.121780 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:18.121664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66b4d4d798-wssc6" event={"ID":"67a217b5-fc9b-4eb4-ac95-d68b96579ded","Type":"ContainerStarted","Data":"bf8d1f97dc98e2b8af3a011488811b253523fdcd13812a9309524f74e8316eff"}
Sep 17 17:35:18.148248 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:18.148197    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66b4d4d798-wssc6" podStartSLOduration=33.148172691 podStartE2EDuration="33.148172691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:18.147658642 +0000 UTC m=+89.514790524" watchObservedRunningTime="2026-09-17 17:35:18.148172691 +0000 UTC m=+89.515304562"
Sep 17 17:35:18.649338 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:18.649302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:18.652002 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:18.651976    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:19.125161 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:19.125135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:19.126466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:19.126440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66b4d4d798-wssc6"
Sep 17 17:35:24.665550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.665513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:24.668154 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.668123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/841de304-78b6-4f84-8b77-ff547166887d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6jkw\" (UID: \"841de304-78b6-4f84-8b77-ff547166887d\") " pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:24.750466 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.750424    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" podUID="bf159bf2-d28f-4574-a08b-b2b389793fa0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:35:24.750590 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.750512    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx"
Sep 17 17:35:24.751049 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.751010    2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1a2bbe00f51d2a477b8a173cfc14a4b7d94c597dc23cc5f5b1bca003458d3b6e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 17:35:24.751173 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.751080    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" podUID="bf159bf2-d28f-4574-a08b-b2b389793fa0" containerName="service-proxy" containerID="cri-o://1a2bbe00f51d2a477b8a173cfc14a4b7d94c597dc23cc5f5b1bca003458d3b6e" gracePeriod=30
Sep 17 17:35:24.944450 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:24.944427    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r6jkw"
Sep 17 17:35:25.145399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:25.145365    2575 generic.go:356] "Generic (PLEG): container finished" podID="bf159bf2-d28f-4574-a08b-b2b389793fa0" containerID="1a2bbe00f51d2a477b8a173cfc14a4b7d94c597dc23cc5f5b1bca003458d3b6e" exitCode=2
Sep 17 17:35:25.145524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:25.145436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerDied","Data":"1a2bbe00f51d2a477b8a173cfc14a4b7d94c597dc23cc5f5b1bca003458d3b6e"}
Sep 17 17:35:25.145524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:25.145484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-588f99bb8d-qmfxx" event={"ID":"bf159bf2-d28f-4574-a08b-b2b389793fa0","Type":"ContainerStarted","Data":"63241ded55772d685cae966aebd0a139e0f30fcd38320e419032c48be2c3ecf1"}
Sep 17 17:35:25.219975 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:25.219914    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r6jkw"]
Sep 17 17:35:25.222750 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:25.222727    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841de304_78b6_4f84_8b77_ff547166887d.slice/crio-d7ce14e2fd0ab799a686a6a37427a0291ecc1b3ded3e5e737ce653b782f4d81f WatchSource:0}: Error finding container d7ce14e2fd0ab799a686a6a37427a0291ecc1b3ded3e5e737ce653b782f4d81f: Status 404 returned error can't find the container with id d7ce14e2fd0ab799a686a6a37427a0291ecc1b3ded3e5e737ce653b782f4d81f
Sep 17 17:35:26.150369 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:26.150299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6jkw" event={"ID":"841de304-78b6-4f84-8b77-ff547166887d","Type":"ContainerStarted","Data":"1f99bc6adaf21c2d10aac873ef2a8704394b0952f7e75ff6969e180273e1413f"}
Sep 17 17:35:26.150696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:26.150412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6jkw" event={"ID":"841de304-78b6-4f84-8b77-ff547166887d","Type":"ContainerStarted","Data":"ff9bb73ccb11d0982e221039a28f812fe85a9f293380c94427b5f22b92f26843"}
Sep 17 17:35:26.150696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:26.150451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6jkw" event={"ID":"841de304-78b6-4f84-8b77-ff547166887d","Type":"ContainerStarted","Data":"d7ce14e2fd0ab799a686a6a37427a0291ecc1b3ded3e5e737ce653b782f4d81f"}
Sep 17 17:35:27.155426 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.155397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6jkw" event={"ID":"841de304-78b6-4f84-8b77-ff547166887d","Type":"ContainerStarted","Data":"54702566cbb042c756f6fa4db09f9621881c5031c6250832cf7a8ac9f645acb6"}
Sep 17 17:35:27.182822 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.182775    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r6jkw" podStartSLOduration=33.525250556 podStartE2EDuration="35.182760156s" totalImagesPullingTime="1.6575096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:27.182084841 +0000 UTC m=+98.549216723" watchObservedRunningTime="2026-09-17 17:35:27.182760156 +0000 UTC m=+98.549892036"
Sep 17 17:35:27.689680 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.689648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:35:27.692060 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.692030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"image-registry-7f5f859bc4-kz5gr\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") " pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:35:27.741955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.741896    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jxpgb\""
Sep 17 17:35:27.750007 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.749979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:35:27.790233 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.790203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:35:27.792353 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:27.792326    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1773536c-31a2-436a-9f79-c5441b182f23-metrics-tls\") pod \"dns-default-88sj8\" (UID: \"1773536c-31a2-436a-9f79-c5441b182f23\") " pod="openshift-dns/dns-default-88sj8"
Sep 17 17:35:28.022465 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.022406    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:35:28.025269 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:28.025233    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9268b1_4cea_4e84_8295_0d0df76b4df4.slice/crio-ee1e3c8ecbfa927cf0497ac800c61f54d9225560e33579ad0be65f8b3e9ed3b9 WatchSource:0}: Error finding container ee1e3c8ecbfa927cf0497ac800c61f54d9225560e33579ad0be65f8b3e9ed3b9: Status 404 returned error can't find the container with id ee1e3c8ecbfa927cf0497ac800c61f54d9225560e33579ad0be65f8b3e9ed3b9
Sep 17 17:35:28.065569 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.065550    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bd5nq\""
Sep 17 17:35:28.073473 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.073457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-88sj8"
Sep 17 17:35:28.160077 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.160041    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" event={"ID":"0a9268b1-4cea-4e84-8295-0d0df76b4df4","Type":"ContainerStarted","Data":"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"}
Sep 17 17:35:28.160489 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.160110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" event={"ID":"0a9268b1-4cea-4e84-8295-0d0df76b4df4","Type":"ContainerStarted","Data":"ee1e3c8ecbfa927cf0497ac800c61f54d9225560e33579ad0be65f8b3e9ed3b9"}
Sep 17 17:35:28.160489 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.160350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:35:28.181578 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.181533    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" podStartSLOduration=97.181519416 podStartE2EDuration="1m37.181519416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:28.18080161 +0000 UTC m=+99.547933490" watchObservedRunningTime="2026-09-17 17:35:28.181519416 +0000 UTC m=+99.548651296"
Sep 17 17:35:28.340451 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:28.340425    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-88sj8"]
Sep 17 17:35:28.343580 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:28.343550    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1773536c_31a2_436a_9f79_c5441b182f23.slice/crio-6f7fa642612a49973369b01c16363083b77c91c65fe893e1cce79d7a03ef8e77 WatchSource:0}: Error finding container 6f7fa642612a49973369b01c16363083b77c91c65fe893e1cce79d7a03ef8e77: Status 404 returned error can't find the container with id 6f7fa642612a49973369b01c16363083b77c91c65fe893e1cce79d7a03ef8e77
Sep 17 17:35:29.166273 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.166224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88sj8" event={"ID":"1773536c-31a2-436a-9f79-c5441b182f23","Type":"ContainerStarted","Data":"6f7fa642612a49973369b01c16363083b77c91c65fe893e1cce79d7a03ef8e77"}
Sep 17 17:35:29.787490 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.787455    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"]
Sep 17 17:35:29.789436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.789406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:29.794147 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.794113    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:35:29.795368 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.795224    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:35:29.795368 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.795224    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:35:29.795368 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.795327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:35:29.795615 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.795328    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n6j8j\""
Sep 17 17:35:29.795615 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.795261    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:35:29.807597 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.807575    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"]
Sep 17 17:35:29.875938 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.875916    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qrtxz"]
Sep 17 17:35:29.877670 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.877655    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:29.879911 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.879886    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:35:29.880247 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.880230    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m5jv7\""
Sep 17 17:35:29.880644 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.880627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:35:29.880718 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.880650    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:35:29.906989 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.906967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:29.907124 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.907008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:29.907124 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.907086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvjs\" (UniqueName: \"kubernetes.io/projected/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-kube-api-access-jtvjs\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:29.907239 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:29.907161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.007647 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-textfile\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007650    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-metrics-client-ca\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq49b\" (UniqueName: \"kubernetes.io/projected/319e8629-d769-4f2e-a130-da999bcf0a07-kube-api-access-rq49b\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007751    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.007942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.007942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-wtmp\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-sys\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.007942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.008085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.008085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.007990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvjs\" (UniqueName: \"kubernetes.io/projected/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-kube-api-access-jtvjs\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.008085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.008043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.008085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.008077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-root\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.008464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.008444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.010032 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.010011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.010254 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.010238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.018634 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.018615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvjs\" (UniqueName: \"kubernetes.io/projected/5f363bc4-50aa-4bb9-9ff4-3ce9426326bd-kube-api-access-jtvjs\") pod \"openshift-state-metrics-86c5bc79d6-g9rkc\" (UID: \"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.098805 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.098782    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"
Sep 17 17:35:30.109046 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109109 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-root\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109109 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-textfile\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109210 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-metrics-client-ca\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109210 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq49b\" (UniqueName: \"kubernetes.io/projected/319e8629-d769-4f2e-a130-da999bcf0a07-kube-api-access-rq49b\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109210 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109210 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-root\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:30.109215    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-wtmp\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-sys\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:35:30.109309    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls podName:319e8629-d769-4f2e-a130-da999bcf0a07 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:30.609259894 +0000 UTC m=+101.976391765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls") pod "node-exporter-qrtxz" (UID: "319e8629-d769-4f2e-a130-da999bcf0a07") : secret "node-exporter-tls" not found
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-sys\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109684 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-textfile\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109684 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-wtmp\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-metrics-client-ca\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.109959 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.109901    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.111388 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.111369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.126676 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.126657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq49b\" (UniqueName: \"kubernetes.io/projected/319e8629-d769-4f2e-a130-da999bcf0a07-kube-api-access-rq49b\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.172787 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.172759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88sj8" event={"ID":"1773536c-31a2-436a-9f79-c5441b182f23","Type":"ContainerStarted","Data":"68198447f1642d37bdffd9b1137345d3c6af5753971c9b97a013655d4688af5d"}
Sep 17 17:35:30.173177 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.172807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-88sj8" event={"ID":"1773536c-31a2-436a-9f79-c5441b182f23","Type":"ContainerStarted","Data":"cc9d7104f7ab473bd718913822c3fa1bf185e0b2334239a8e08747b24fbba668"}
Sep 17 17:35:30.173177 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.172873    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-88sj8"
Sep 17 17:35:30.191364 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.191316    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-88sj8" podStartSLOduration=65.736305316 podStartE2EDuration="1m7.191300705s" totalImagesPullingTime="1.454995389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:30.190305958 +0000 UTC m=+101.557437840" watchObservedRunningTime="2026-09-17 17:35:30.191300705 +0000 UTC m=+101.558432585"
Sep 17 17:35:30.374513 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.374459    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc"]
Sep 17 17:35:30.377479 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:30.377456    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f363bc4_50aa_4bb9_9ff4_3ce9426326bd.slice/crio-02cb78c3cda0ac8aeb7730daabfe69aacad0cb7b4e19a68784dc40da5b2146b3 WatchSource:0}: Error finding container 02cb78c3cda0ac8aeb7730daabfe69aacad0cb7b4e19a68784dc40da5b2146b3: Status 404 returned error can't find the container with id 02cb78c3cda0ac8aeb7730daabfe69aacad0cb7b4e19a68784dc40da5b2146b3
Sep 17 17:35:30.614436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.614404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.616599 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.616580    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/319e8629-d769-4f2e-a130-da999bcf0a07-node-exporter-tls\") pod \"node-exporter-qrtxz\" (UID: \"319e8629-d769-4f2e-a130-da999bcf0a07\") " pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.787204 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:30.787142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrtxz"
Sep 17 17:35:30.799283 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:30.799261    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319e8629_d769_4f2e_a130_da999bcf0a07.slice/crio-03e616e25ef8d3cb7e8086863344b071478264972ae9806eb96b114013902bef WatchSource:0}: Error finding container 03e616e25ef8d3cb7e8086863344b071478264972ae9806eb96b114013902bef: Status 404 returned error can't find the container with id 03e616e25ef8d3cb7e8086863344b071478264972ae9806eb96b114013902bef
Sep 17 17:35:31.177997 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:31.177964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc" event={"ID":"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd","Type":"ContainerStarted","Data":"a5e6777603b9ecd7cd66f84dc461ea86e2e7b3c4e4b4a66b159111dfab85bbc2"}
Sep 17 17:35:31.178402 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:31.178032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc" event={"ID":"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd","Type":"ContainerStarted","Data":"94126e835d85f098a5666c3b777866b545ea05e102a0f92f279b120fcdd65610"}
Sep 17 17:35:31.178402 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:31.178063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc" event={"ID":"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd","Type":"ContainerStarted","Data":"02cb78c3cda0ac8aeb7730daabfe69aacad0cb7b4e19a68784dc40da5b2146b3"}
Sep 17 17:35:31.179276 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:31.179250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrtxz" event={"ID":"319e8629-d769-4f2e-a130-da999bcf0a07","Type":"ContainerStarted","Data":"03e616e25ef8d3cb7e8086863344b071478264972ae9806eb96b114013902bef"}
Sep 17 17:35:32.185165 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:32.185134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc" event={"ID":"5f363bc4-50aa-4bb9-9ff4-3ce9426326bd","Type":"ContainerStarted","Data":"9731d9413b953773c1d2785888b423c873160cf9f600024c7d57a1b2ffd3182d"}
Sep 17 17:35:32.209208 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:32.209167    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-g9rkc" podStartSLOduration=1.6213755779999999 podStartE2EDuration="3.209149385s" totalImagesPullingTime="1.587773807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:32.209082975 +0000 UTC m=+103.576214856" watchObservedRunningTime="2026-09-17 17:35:32.209149385 +0000 UTC m=+103.576281267"
Sep 17 17:35:33.190864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.190824    2575 generic.go:356] "Generic (PLEG): container finished" podID="319e8629-d769-4f2e-a130-da999bcf0a07" containerID="e74e1295087bc5f5b50f1ad6d18133687ed4e6cf8f69c624c28445700e122d83" exitCode=0
Sep 17 17:35:33.191308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.190926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrtxz" event={"ID":"319e8629-d769-4f2e-a130-da999bcf0a07","Type":"ContainerDied","Data":"e74e1295087bc5f5b50f1ad6d18133687ed4e6cf8f69c624c28445700e122d83"}
Sep 17 17:35:33.969356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.969324    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:35:33.972049 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.972032    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:33.974578 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.974551    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:35:33.974578 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.974551    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:35:33.974735 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.974694    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:35:33.974773 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.974694    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:35:33.975805 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.975778    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:35:33.975925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.975785    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vkgxj\""
Sep 17 17:35:33.975925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.975909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:35:33.976078 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.976060    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:35:33.979865 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.979843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:35:33.983644 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:33.983588    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:35:34.141525 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141637 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141637 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpj8\" (UniqueName: \"kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141637 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141819 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141819 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.141819 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.141744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.197666 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.197636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrtxz" event={"ID":"319e8629-d769-4f2e-a130-da999bcf0a07","Type":"ContainerStarted","Data":"7f2e6d312d40d95d5e319652d2800e0cf1a5286870cd1e07f34783ad117f44c0"}
Sep 17 17:35:34.197982 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.197687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrtxz" event={"ID":"319e8629-d769-4f2e-a130-da999bcf0a07","Type":"ContainerStarted","Data":"b37cef3ccb627bd3776fcfba7a43223f042dd1e75b0eb196723d60c94afdf9a1"}
Sep 17 17:35:34.225838 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.225761    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qrtxz" podStartSLOduration=3.539939869 podStartE2EDuration="5.225748482s" totalImagesPullingTime="1.601914847s" totalInitContainerRuntime="83.893766ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:34.224876144 +0000 UTC m=+105.592008025" watchObservedRunningTime="2026-09-17 17:35:34.225748482 +0000 UTC m=+105.592880362"
Sep 17 17:35:34.242254 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242232    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242374 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242591 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242591 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.242591 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.242459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpj8\" (UniqueName: \"kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.243179 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.243152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.243179 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.243178    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.243403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.243382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.243470 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.243390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.244943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.244920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.245040 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.244959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.253449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.253431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpj8\" (UniqueName: \"kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8\") pod \"console-565c56fc49-zmz6k\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") " pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.281349 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.281330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:34.567550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:34.567471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:35:34.571866 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:34.571845    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2698877_38c8_44df_92ba_bf7074ea0ce9.slice/crio-553d5de6aba915508816b7b37e19bb579ce580cc31283411da07d441fa99cc7c WatchSource:0}: Error finding container 553d5de6aba915508816b7b37e19bb579ce580cc31283411da07d441fa99cc7c: Status 404 returned error can't find the container with id 553d5de6aba915508816b7b37e19bb579ce580cc31283411da07d441fa99cc7c
Sep 17 17:35:35.203697 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:35.203659    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c56fc49-zmz6k" event={"ID":"e2698877-38c8-44df-92ba-bf7074ea0ce9","Type":"ContainerStarted","Data":"553d5de6aba915508816b7b37e19bb579ce580cc31283411da07d441fa99cc7c"}
Sep 17 17:35:38.429018 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.428987    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:35:38.432031 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.432015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.445853 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.445831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:35:38.578355 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2vx\" (UniqueName: \"kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578609 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578609 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.578609 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.578542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.679930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2vx\" (UniqueName: \"kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.679930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.679930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.679930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.679986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.680008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680736 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.680715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.680780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.680867 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.680842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.681074 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.681049    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.682396 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.682378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.682689 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.682670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.692017 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.691993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2vx\" (UniqueName: \"kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx\") pod \"console-cdf68bb9-9dtr9\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") " pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:38.743228 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:38.743212    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:39.012576 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.012509    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:35:39.016599 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:35:39.016570    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd6adb4_494f_4fd7_af11_e0bac5fe2eba.slice/crio-c9427bfc7b88d5441a865cc640443f6fbc84679cb8f47a0c7979f21b67461a02 WatchSource:0}: Error finding container c9427bfc7b88d5441a865cc640443f6fbc84679cb8f47a0c7979f21b67461a02: Status 404 returned error can't find the container with id c9427bfc7b88d5441a865cc640443f6fbc84679cb8f47a0c7979f21b67461a02
Sep 17 17:35:39.219848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.219813    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c56fc49-zmz6k" event={"ID":"e2698877-38c8-44df-92ba-bf7074ea0ce9","Type":"ContainerStarted","Data":"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"}
Sep 17 17:35:39.221502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.221475    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdf68bb9-9dtr9" event={"ID":"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba","Type":"ContainerStarted","Data":"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"}
Sep 17 17:35:39.221618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.221517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdf68bb9-9dtr9" event={"ID":"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba","Type":"ContainerStarted","Data":"c9427bfc7b88d5441a865cc640443f6fbc84679cb8f47a0c7979f21b67461a02"}
Sep 17 17:35:39.245719 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.245672    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-565c56fc49-zmz6k" podStartSLOduration=2.363449702 podStartE2EDuration="6.245658216s" totalImagesPullingTime="3.882208514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:39.24460902 +0000 UTC m=+110.611740892" watchObservedRunningTime="2026-09-17 17:35:39.245658216 +0000 UTC m=+110.612790097"
Sep 17 17:35:39.264979 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:39.264907    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cdf68bb9-9dtr9" podStartSLOduration=1.2648929039999999 podStartE2EDuration="1.264892904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:39.263543608 +0000 UTC m=+110.630675488" watchObservedRunningTime="2026-09-17 17:35:39.264892904 +0000 UTC m=+110.632024785"
Sep 17 17:35:40.181960 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:40.181921    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-88sj8"
Sep 17 17:35:40.487090 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:40.487020    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:35:41.811931 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:41.811894    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:35:44.282188 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:44.282154    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:35:48.743953 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:48.743910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:48.744517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:48.743994    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:48.748702 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:48.748662    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:49.259756 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:49.259732    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:35:50.493132 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:50.493081    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:35:52.669736 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:35:52.669701    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:36:00.627163 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:00.627124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:36:00.629446 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:00.629423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aff5a10-19db-45d6-8463-2fdfea258bbf-metrics-certs\") pod \"network-metrics-daemon-twm78\" (UID: \"2aff5a10-19db-45d6-8463-2fdfea258bbf\") " pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:36:00.696806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:00.696782    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qcgkp\""
Sep 17 17:36:00.704924 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:00.704905    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm78"
Sep 17 17:36:00.979248 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:00.979179    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twm78"]
Sep 17 17:36:00.982811 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:36:00.982785    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aff5a10_19db_45d6_8463_2fdfea258bbf.slice/crio-34586584f0b3988c91dfb7d108eb0501c9ccc656f1fb223d49473ce9c1bfdc2c WatchSource:0}: Error finding container 34586584f0b3988c91dfb7d108eb0501c9ccc656f1fb223d49473ce9c1bfdc2c: Status 404 returned error can't find the container with id 34586584f0b3988c91dfb7d108eb0501c9ccc656f1fb223d49473ce9c1bfdc2c
Sep 17 17:36:01.305149 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:01.305067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm78" event={"ID":"2aff5a10-19db-45d6-8463-2fdfea258bbf","Type":"ContainerStarted","Data":"34586584f0b3988c91dfb7d108eb0501c9ccc656f1fb223d49473ce9c1bfdc2c"}
Sep 17 17:36:02.311221 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:02.311149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm78" event={"ID":"2aff5a10-19db-45d6-8463-2fdfea258bbf","Type":"ContainerStarted","Data":"c61e93a108375458de04ef49438d7e3a71afd0729a29ccde5cc3dadf65da4d6d"}
Sep 17 17:36:02.311542 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:02.311236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm78" event={"ID":"2aff5a10-19db-45d6-8463-2fdfea258bbf","Type":"ContainerStarted","Data":"998ef33ff91fc12f403281344f9f41a53261329dada79c01d053f82cfe08410a"}
Sep 17 17:36:02.341706 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:02.341664    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-twm78" podStartSLOduration=130.451917455 podStartE2EDuration="2m11.34165103s" totalImagesPullingTime="889.733575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:02.340885868 +0000 UTC m=+133.708017750" watchObservedRunningTime="2026-09-17 17:36:02.34165103 +0000 UTC m=+133.708782910"
Sep 17 17:36:05.505394 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.505343    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" podUID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" containerName="registry" containerID="cri-o://3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1" gracePeriod=30
Sep 17 17:36:05.743411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.743387    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:36:05.760134 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760065    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760176    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760259    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj76p\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760333    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca\") pod \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\" (UID: \"0a9268b1-4cea-4e84-8295-0d0df76b4df4\") "
Sep 17 17:36:05.760574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760552    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:05.760870 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.760837    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:05.762951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.762906    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:05.762951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.762931    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:05.763153 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.763015    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:05.763153 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.763083    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p" (OuterVolumeSpecName: "kube-api-access-kj76p") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "kube-api-access-kj76p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:05.763153 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.763143    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:05.769762 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.769738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0a9268b1-4cea-4e84-8295-0d0df76b4df4" (UID: "0a9268b1-4cea-4e84-8295-0d0df76b4df4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:05.861125 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861086    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-trusted-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861125 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861124    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a9268b1-4cea-4e84-8295-0d0df76b4df4-ca-trust-extracted\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861134    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-image-registry-private-configuration\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861145    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-certificates\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861154    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-bound-sa-token\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861163    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a9268b1-4cea-4e84-8295-0d0df76b4df4-installation-pull-secrets\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861171    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj76p\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-kube-api-access-kj76p\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.861310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:05.861179    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a9268b1-4cea-4e84-8295-0d0df76b4df4-registry-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:06.325949 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.325919    2575 generic.go:356] "Generic (PLEG): container finished" podID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" containerID="3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1" exitCode=0
Sep 17 17:36:06.326055 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.325970    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr"
Sep 17 17:36:06.326055 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.325988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" event={"ID":"0a9268b1-4cea-4e84-8295-0d0df76b4df4","Type":"ContainerDied","Data":"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"}
Sep 17 17:36:06.326055 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.326030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5f859bc4-kz5gr" event={"ID":"0a9268b1-4cea-4e84-8295-0d0df76b4df4","Type":"ContainerDied","Data":"ee1e3c8ecbfa927cf0497ac800c61f54d9225560e33579ad0be65f8b3e9ed3b9"}
Sep 17 17:36:06.326055 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.326050    2575 scope.go:117] "RemoveContainer" containerID="3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"
Sep 17 17:36:06.334817 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.334799    2575 scope.go:117] "RemoveContainer" containerID="3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"
Sep 17 17:36:06.335084 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:36:06.335064    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1\": container with ID starting with 3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1 not found: ID does not exist" containerID="3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"
Sep 17 17:36:06.335152 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.335091    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1"} err="failed to get container status \"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1\": rpc error: code = NotFound desc = could not find container \"3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1\": container with ID starting with 3a99fe846ebb6a25a3e6fdf0a5953c819f5c23ea325dd023ea1d9f0eaabaa9c1 not found: ID does not exist"
Sep 17 17:36:06.354723 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.354698    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:36:06.358442 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.358420    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f5f859bc4-kz5gr"]
Sep 17 17:36:06.777744 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.777699    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" path="/var/lib/kubelet/pods/0a9268b1-4cea-4e84-8295-0d0df76b4df4/volumes"
Sep 17 17:36:06.831648 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:06.831599    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-565c56fc49-zmz6k" podUID="e2698877-38c8-44df-92ba-bf7074ea0ce9" containerName="console" containerID="cri-o://067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf" gracePeriod=15
Sep 17 17:36:07.051687 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.051666    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:36:07.068738 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068718    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.068840 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068748    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.068840 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068792    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpj8\" (UniqueName: \"kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.068947 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068907    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.069001 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068943    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.069001 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.068985    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.069001 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069000    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config\") pod \"e2698877-38c8-44df-92ba-bf7074ea0ce9\" (UID: \"e2698877-38c8-44df-92ba-bf7074ea0ce9\") "
Sep 17 17:36:07.069166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069078    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.069222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069199    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.069354 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069231    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-oauth-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.069427 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069411    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config" (OuterVolumeSpecName: "console-config") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.069489 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.069467    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.071044 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.071023    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:07.071172 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.071154    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:07.071229 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.071207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8" (OuterVolumeSpecName: "kube-api-access-btpj8") pod "e2698877-38c8-44df-92ba-bf7074ea0ce9" (UID: "e2698877-38c8-44df-92ba-bf7074ea0ce9"). InnerVolumeSpecName "kube-api-access-btpj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:07.170075 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170044    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-oauth-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.170075 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170074    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.170075 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170084    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-trusted-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.170246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170109    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpj8\" (UniqueName: \"kubernetes.io/projected/e2698877-38c8-44df-92ba-bf7074ea0ce9-kube-api-access-btpj8\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.170246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170124    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2698877-38c8-44df-92ba-bf7074ea0ce9-console-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.170246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.170137    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2698877-38c8-44df-92ba-bf7074ea0ce9-service-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.333821 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.333786    2575 generic.go:356] "Generic (PLEG): container finished" podID="e2698877-38c8-44df-92ba-bf7074ea0ce9" containerID="067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf" exitCode=0
Sep 17 17:36:07.333961 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.333842    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565c56fc49-zmz6k"
Sep 17 17:36:07.333961 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.333890    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c56fc49-zmz6k" event={"ID":"e2698877-38c8-44df-92ba-bf7074ea0ce9","Type":"ContainerDied","Data":"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"}
Sep 17 17:36:07.334054 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.334002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565c56fc49-zmz6k" event={"ID":"e2698877-38c8-44df-92ba-bf7074ea0ce9","Type":"ContainerDied","Data":"553d5de6aba915508816b7b37e19bb579ce580cc31283411da07d441fa99cc7c"}
Sep 17 17:36:07.334054 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.334036    2575 scope.go:117] "RemoveContainer" containerID="067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"
Sep 17 17:36:07.343767 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.343749    2575 scope.go:117] "RemoveContainer" containerID="067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"
Sep 17 17:36:07.344063 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:36:07.344041    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf\": container with ID starting with 067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf not found: ID does not exist" containerID="067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"
Sep 17 17:36:07.344116 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.344073    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf"} err="failed to get container status \"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf\": rpc error: code = NotFound desc = could not find container \"067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf\": container with ID starting with 067971a97dc3d6047f76a560d0a7b013d956cadd707af80482770a7083102edf not found: ID does not exist"
Sep 17 17:36:07.371273 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.371248    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:36:07.379483 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:07.379460    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-565c56fc49-zmz6k"]
Sep 17 17:36:08.776527 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:08.776493    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2698877-38c8-44df-92ba-bf7074ea0ce9" path="/var/lib/kubelet/pods/e2698877-38c8-44df-92ba-bf7074ea0ce9/volumes"
Sep 17 17:36:17.688114 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.688049    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cdf68bb9-9dtr9" podUID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" containerName="console" containerID="cri-o://e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd" gracePeriod=15
Sep 17 17:36:17.913177 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.913154    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:36:17.938005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.937979    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938164 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938030    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938164 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938156    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938292 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938185    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938292 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938292 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938248    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938292 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938267    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2vx\" (UniqueName: \"kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx\") pod \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\" (UID: \"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba\") "
Sep 17 17:36:17.938491 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca" (OuterVolumeSpecName: "service-ca") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:17.938553 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:17.938553 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938508    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config" (OuterVolumeSpecName: "console-config") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:17.938794 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.938772    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:17.940174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.940148    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:17.940385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.940358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx" (OuterVolumeSpecName: "kube-api-access-zs2vx") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "kube-api-access-zs2vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:17.940667 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:17.940643    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" (UID: "1bd6adb4-494f-4fd7-af11-e0bac5fe2eba"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:18.038796 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038779    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-service-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038796 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038798    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-trusted-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038807    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038815    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-oauth-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038824    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038833    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-console-oauth-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.038910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.038841    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2vx\" (UniqueName: \"kubernetes.io/projected/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba-kube-api-access-zs2vx\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:36:18.374675 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.374646    2575 generic.go:356] "Generic (PLEG): container finished" podID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" containerID="e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd" exitCode=0
Sep 17 17:36:18.374778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.374695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdf68bb9-9dtr9" event={"ID":"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba","Type":"ContainerDied","Data":"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"}
Sep 17 17:36:18.374778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.374740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdf68bb9-9dtr9" event={"ID":"1bd6adb4-494f-4fd7-af11-e0bac5fe2eba","Type":"ContainerDied","Data":"c9427bfc7b88d5441a865cc640443f6fbc84679cb8f47a0c7979f21b67461a02"}
Sep 17 17:36:18.374778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.374762    2575 scope.go:117] "RemoveContainer" containerID="e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"
Sep 17 17:36:18.374778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.374700    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdf68bb9-9dtr9"
Sep 17 17:36:18.384140 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.384116    2575 scope.go:117] "RemoveContainer" containerID="e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"
Sep 17 17:36:18.384464 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:36:18.384438    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd\": container with ID starting with e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd not found: ID does not exist" containerID="e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"
Sep 17 17:36:18.384533 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.384474    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd"} err="failed to get container status \"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd\": rpc error: code = NotFound desc = could not find container \"e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd\": container with ID starting with e98d6f47020783e9c3ca8baf42cc063f43eaf8ac48fb273218e7aa58dad8d0dd not found: ID does not exist"
Sep 17 17:36:18.398344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.398307    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:36:18.402820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.402797    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cdf68bb9-9dtr9"]
Sep 17 17:36:18.776711 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:18.776633    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" path="/var/lib/kubelet/pods/1bd6adb4-494f-4fd7-af11-e0bac5fe2eba/volumes"
Sep 17 17:36:54.821442 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821406    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821732    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2698877-38c8-44df-92ba-bf7074ea0ce9" containerName="console"
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821748    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2698877-38c8-44df-92ba-bf7074ea0ce9" containerName="console"
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821774    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" containerName="console"
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821783    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" containerName="console"
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821802    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" containerName="registry"
Sep 17 17:36:54.821859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821811    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" containerName="registry"
Sep 17 17:36:54.822085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821866    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9268b1-4cea-4e84-8295-0d0df76b4df4" containerName="registry"
Sep 17 17:36:54.822085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821878    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2698877-38c8-44df-92ba-bf7074ea0ce9" containerName="console"
Sep 17 17:36:54.822085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.821893    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd6adb4-494f-4fd7-af11-e0bac5fe2eba" containerName="console"
Sep 17 17:36:54.826804 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.826781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.829452 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.829361    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:36:54.829571 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.829485    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:36:54.829775 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.829753    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:36:54.829974 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.829945    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:36:54.830032 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.830012    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:36:54.830349 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.830331    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:36:54.830468 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.830375    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vkgxj\""
Sep 17 17:36:54.830468 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.830329    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:36:54.835559 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.835538    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:36:54.843703 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.843675    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:36:54.975902 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.975875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.975908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.975933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.975957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqkt\" (UniqueName: \"kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.976014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.976043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:54.976250 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:54.976058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.076886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.076813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.076886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.076849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.076886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.076865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.076886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.076882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqkt\" (UniqueName: \"kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077634 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077785 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077764    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.077984 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.077909    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.079263 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.079246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.079540 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.079516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.086040 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.086018    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqkt\" (UniqueName: \"kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt\") pod \"console-755946c6bf-hj2wl\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") " pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.136908 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.136885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:36:55.406674 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.406645    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:36:55.409710 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:36:55.409682    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26178025_53b6_489c_a608_c2a08889ab68.slice/crio-651d21a0a80f33b9b4ac0974f2418f4859a4243e7d603fd3632baa2bcf0e464c WatchSource:0}: Error finding container 651d21a0a80f33b9b4ac0974f2418f4859a4243e7d603fd3632baa2bcf0e464c: Status 404 returned error can't find the container with id 651d21a0a80f33b9b4ac0974f2418f4859a4243e7d603fd3632baa2bcf0e464c
Sep 17 17:36:55.510687 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.510656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755946c6bf-hj2wl" event={"ID":"26178025-53b6-489c-a608-c2a08889ab68","Type":"ContainerStarted","Data":"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"}
Sep 17 17:36:55.510815 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.510700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755946c6bf-hj2wl" event={"ID":"26178025-53b6-489c-a608-c2a08889ab68","Type":"ContainerStarted","Data":"651d21a0a80f33b9b4ac0974f2418f4859a4243e7d603fd3632baa2bcf0e464c"}
Sep 17 17:36:55.529520 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:36:55.529476    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755946c6bf-hj2wl" podStartSLOduration=1.5294603740000001 podStartE2EDuration="1.529460374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:55.528594563 +0000 UTC m=+186.895726504" watchObservedRunningTime="2026-09-17 17:36:55.529460374 +0000 UTC m=+186.896592254"
Sep 17 17:37:05.137622 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:37:05.137581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:37:05.138153 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:37:05.137642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:37:05.142384 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:37:05.142356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:37:05.550736 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:37:05.550666    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:38:50.679610 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:38:50.679579    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:38:50.681370 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:38:50.681346    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:38:50.684994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:38:50.684975    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:40:06.634030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.633957    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:06.636987 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.636968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.642944 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.642915    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:40:06.643090 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.642966    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7khhx\""
Sep 17 17:40:06.643188 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.642927    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:40:06.643188 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.643062    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:40:06.644581 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.644559    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"]
Sep 17 17:40:06.647735 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.647717    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.649202 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.649171    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:06.651417 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.651400    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:40:06.651513 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.651459    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-j48hp\""
Sep 17 17:40:06.655952 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.655923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.656057 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.656001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lrt\" (UniqueName: \"kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.657540 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.657451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"]
Sep 17 17:40:06.756991 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.756952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.757203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.757009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8s8\" (UniqueName: \"kubernetes.io/projected/98b7eb32-ce43-463b-8e6b-7f8f29718710-kube-api-access-rb8s8\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.757203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.757029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lrt\" (UniqueName: \"kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.757203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.757063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.759446 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.759414    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.766460 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.766434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lrt\" (UniqueName: \"kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt\") pod \"kserve-controller-manager-9b487cb79-4wmxf\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") " pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:06.858215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.858178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8s8\" (UniqueName: \"kubernetes.io/projected/98b7eb32-ce43-463b-8e6b-7f8f29718710-kube-api-access-rb8s8\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.858403 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.858247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.858403 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:40:06.858367    2575 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 17:40:06.858554 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:40:06.858439    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert podName:98b7eb32-ce43-463b-8e6b-7f8f29718710 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:07.358419034 +0000 UTC m=+378.725550893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert") pod "llmisvc-controller-manager-645f9c8fd6-kv6rc" (UID: "98b7eb32-ce43-463b-8e6b-7f8f29718710") : secret "llmisvc-webhook-server-cert" not found
Sep 17 17:40:06.866930 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.866900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8s8\" (UniqueName: \"kubernetes.io/projected/98b7eb32-ce43-463b-8e6b-7f8f29718710-kube-api-access-rb8s8\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:06.952323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:06.952238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:07.219919 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.219835    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:07.222962 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:40:07.222926    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d64dd4_d39f_4eac_89cb_a0eb0e6b21e6.slice/crio-9aff477b5dbd00ffee08022c6ae245de9e4485db34434890022af425342697b4 WatchSource:0}: Error finding container 9aff477b5dbd00ffee08022c6ae245de9e4485db34434890022af425342697b4: Status 404 returned error can't find the container with id 9aff477b5dbd00ffee08022c6ae245de9e4485db34434890022af425342697b4
Sep 17 17:40:07.224281 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.224263    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:40:07.363475 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.363425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:07.365828 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.365800    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98b7eb32-ce43-463b-8e6b-7f8f29718710-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kv6rc\" (UID: \"98b7eb32-ce43-463b-8e6b-7f8f29718710\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:07.561554 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.561460    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:07.830748 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:07.830723    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"]
Sep 17 17:40:07.833329 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:40:07.833289    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98b7eb32_ce43_463b_8e6b_7f8f29718710.slice/crio-c0b4ade3af6fb9d961fed1577d1886f964c094967527ae299928ff6f4c7c17fd WatchSource:0}: Error finding container c0b4ade3af6fb9d961fed1577d1886f964c094967527ae299928ff6f4c7c17fd: Status 404 returned error can't find the container with id c0b4ade3af6fb9d961fed1577d1886f964c094967527ae299928ff6f4c7c17fd
Sep 17 17:40:08.157750 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:08.157658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" event={"ID":"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6","Type":"ContainerStarted","Data":"9aff477b5dbd00ffee08022c6ae245de9e4485db34434890022af425342697b4"}
Sep 17 17:40:08.159080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:08.159044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc" event={"ID":"98b7eb32-ce43-463b-8e6b-7f8f29718710","Type":"ContainerStarted","Data":"c0b4ade3af6fb9d961fed1577d1886f964c094967527ae299928ff6f4c7c17fd"}
Sep 17 17:40:11.177995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.177963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc" event={"ID":"98b7eb32-ce43-463b-8e6b-7f8f29718710","Type":"ContainerStarted","Data":"9ba74e9b2f1d9af11fbf4c221ec01f0b9d39624eb38222765000fd0ee9de233c"}
Sep 17 17:40:11.178449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.178076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:11.179488 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.179465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" event={"ID":"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6","Type":"ContainerStarted","Data":"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"}
Sep 17 17:40:11.179578 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.179507    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:11.194350 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.194309    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc" podStartSLOduration=2.215693521 podStartE2EDuration="5.194296086s" totalImagesPullingTime="2.978602565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:11.193115613 +0000 UTC m=+382.560247485" watchObservedRunningTime="2026-09-17 17:40:11.194296086 +0000 UTC m=+382.561427966"
Sep 17 17:40:11.208994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:11.208956    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" podStartSLOduration=1.622045286 podStartE2EDuration="5.208944s" totalImagesPullingTime="3.586898714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:11.208133077 +0000 UTC m=+382.575264961" watchObservedRunningTime="2026-09-17 17:40:11.208944 +0000 UTC m=+382.576075881"
Sep 17 17:40:30.052496 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.052463    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f44f854f-p2zzt"]
Sep 17 17:40:30.055634 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.055612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.066301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.066279    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f44f854f-p2zzt"]
Sep 17 17:40:30.137888 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.137854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.137888 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.137888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-trusted-ca-bundle\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.138087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.137975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-service-ca\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.138087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.137997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.138087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.138014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgwn\" (UniqueName: \"kubernetes.io/projected/f919820f-f9bf-4d17-aef1-9b0f01290e3d-kube-api-access-kqgwn\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.138087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.138031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-oauth-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.138246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.138092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-oauth-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.238739 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-oauth-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.238739 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-oauth-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-trusted-ca-bundle\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-service-ca\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.238971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgwn\" (UniqueName: \"kubernetes.io/projected/f919820f-f9bf-4d17-aef1-9b0f01290e3d-kube-api-access-kqgwn\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239702 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.239678    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-oauth-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239823 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.239773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-service-ca\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.239862 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.239837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.240005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.239980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f919820f-f9bf-4d17-aef1-9b0f01290e3d-trusted-ca-bundle\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.241449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.241428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-serving-cert\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.241548 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.241482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f919820f-f9bf-4d17-aef1-9b0f01290e3d-console-oauth-config\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.249257 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.249237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgwn\" (UniqueName: \"kubernetes.io/projected/f919820f-f9bf-4d17-aef1-9b0f01290e3d-kube-api-access-kqgwn\") pod \"console-5f44f854f-p2zzt\" (UID: \"f919820f-f9bf-4d17-aef1-9b0f01290e3d\") " pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.364356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.364321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:30.635581 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:30.635455    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f44f854f-p2zzt"]
Sep 17 17:40:30.637928 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:40:30.637900    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf919820f_f9bf_4d17_aef1_9b0f01290e3d.slice/crio-66ca127b03976c560376cafa0980e09ad59f6a29744a5a28ff78818bbede32d9 WatchSource:0}: Error finding container 66ca127b03976c560376cafa0980e09ad59f6a29744a5a28ff78818bbede32d9: Status 404 returned error can't find the container with id 66ca127b03976c560376cafa0980e09ad59f6a29744a5a28ff78818bbede32d9
Sep 17 17:40:31.249530 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:31.249495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f44f854f-p2zzt" event={"ID":"f919820f-f9bf-4d17-aef1-9b0f01290e3d","Type":"ContainerStarted","Data":"02fb0c39256476956b5d48bfec9c42e648eae6bf1eb9f04ce8fc1c0d43eda449"}
Sep 17 17:40:31.249876 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:31.249542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f44f854f-p2zzt" event={"ID":"f919820f-f9bf-4d17-aef1-9b0f01290e3d","Type":"ContainerStarted","Data":"66ca127b03976c560376cafa0980e09ad59f6a29744a5a28ff78818bbede32d9"}
Sep 17 17:40:31.269759 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:31.269710    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f44f854f-p2zzt" podStartSLOduration=1.2696914750000001 podStartE2EDuration="1.269691475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:31.267372577 +0000 UTC m=+402.634504457" watchObservedRunningTime="2026-09-17 17:40:31.269691475 +0000 UTC m=+402.636823355"
Sep 17 17:40:40.365080 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:40.365046    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:40.365702 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:40.365124    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:40.369828 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:40.369803    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:41.288303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:41.288279    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f44f854f-p2zzt"
Sep 17 17:40:41.334552 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:41.334524    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:40:42.185118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:42.185062    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kv6rc"
Sep 17 17:40:42.188004 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:42.187978    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:49.581334 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.581295    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:49.581812 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.581566    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" podUID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" containerName="manager" containerID="cri-o://0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f" gracePeriod=10
Sep 17 17:40:49.602334 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.602302    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-cmxb5"]
Sep 17 17:40:49.604663 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.604644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.616499 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.616475    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-cmxb5"]
Sep 17 17:40:49.688542 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.688517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/4106a474-e143-40c1-bbfe-be7abf40ba54-kube-api-access-tr9hq\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.688660 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.688550    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4106a474-e143-40c1-bbfe-be7abf40ba54-cert\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.789342 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.789313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/4106a474-e143-40c1-bbfe-be7abf40ba54-kube-api-access-tr9hq\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.789532 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.789350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4106a474-e143-40c1-bbfe-be7abf40ba54-cert\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.791695 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.791672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4106a474-e143-40c1-bbfe-be7abf40ba54-cert\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.798629 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.798603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/4106a474-e143-40c1-bbfe-be7abf40ba54-kube-api-access-tr9hq\") pod \"kserve-controller-manager-9b487cb79-cmxb5\" (UID: \"4106a474-e143-40c1-bbfe-be7abf40ba54\") " pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.810328 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.810310    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:49.889968 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.889899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4lrt\" (UniqueName: \"kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt\") pod \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") "
Sep 17 17:40:49.889968 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.889938    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert\") pod \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\" (UID: \"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6\") "
Sep 17 17:40:49.891884 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.891859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert" (OuterVolumeSpecName: "cert") pod "b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" (UID: "b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:49.892014 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.891995    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt" (OuterVolumeSpecName: "kube-api-access-v4lrt") pod "b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" (UID: "b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6"). InnerVolumeSpecName "kube-api-access-v4lrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:40:49.947058 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.947025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:49.990888 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.990861    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lrt\" (UniqueName: \"kubernetes.io/projected/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-kube-api-access-v4lrt\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:40:49.990964 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:49.990897    2575 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:40:50.214474 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.214443    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-cmxb5"]
Sep 17 17:40:50.216723 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:40:50.216697    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4106a474_e143_40c1_bbfe_be7abf40ba54.slice/crio-8df6a8fe6b4509b3076050a0db3d89a1d57217e4bc930cdc633dad557a92cec6 WatchSource:0}: Error finding container 8df6a8fe6b4509b3076050a0db3d89a1d57217e4bc930cdc633dad557a92cec6: Status 404 returned error can't find the container with id 8df6a8fe6b4509b3076050a0db3d89a1d57217e4bc930cdc633dad557a92cec6
Sep 17 17:40:50.316788 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.316749    2575 generic.go:356] "Generic (PLEG): container finished" podID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" containerID="0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f" exitCode=0
Sep 17 17:40:50.316943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.316795    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf"
Sep 17 17:40:50.316943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.316837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" event={"ID":"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6","Type":"ContainerDied","Data":"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"}
Sep 17 17:40:50.316943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.316903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-4wmxf" event={"ID":"b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6","Type":"ContainerDied","Data":"9aff477b5dbd00ffee08022c6ae245de9e4485db34434890022af425342697b4"}
Sep 17 17:40:50.316943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.316932    2575 scope.go:117] "RemoveContainer" containerID="0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"
Sep 17 17:40:50.318075 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.318034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5" event={"ID":"4106a474-e143-40c1-bbfe-be7abf40ba54","Type":"ContainerStarted","Data":"8df6a8fe6b4509b3076050a0db3d89a1d57217e4bc930cdc633dad557a92cec6"}
Sep 17 17:40:50.324763 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.324703    2575 scope.go:117] "RemoveContainer" containerID="0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"
Sep 17 17:40:50.325052 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:40:50.325032    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f\": container with ID starting with 0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f not found: ID does not exist" containerID="0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"
Sep 17 17:40:50.325143 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.325059    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f"} err="failed to get container status \"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f\": rpc error: code = NotFound desc = could not find container \"0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f\": container with ID starting with 0d73ff9ca3da513ad5ee39a221de6487ef357db6d879e49197c2ec47a0e6812f not found: ID does not exist"
Sep 17 17:40:50.339227 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.339207    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:50.343613 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.343595    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-4wmxf"]
Sep 17 17:40:50.778078 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:50.778037    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" path="/var/lib/kubelet/pods/b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6/volumes"
Sep 17 17:40:51.322973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:51.322942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5" event={"ID":"4106a474-e143-40c1-bbfe-be7abf40ba54","Type":"ContainerStarted","Data":"95e48a3e270016ddaf3e6986b79787eddfe8337cea73e191d043f81c6b3411d8"}
Sep 17 17:40:51.323166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:51.323052    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:40:51.341524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:40:51.341479    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5" podStartSLOduration=1.940640275 podStartE2EDuration="2.341466269s" totalImagesPullingTime="400.825994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:51.339570136 +0000 UTC m=+422.706702016" watchObservedRunningTime="2026-09-17 17:40:51.341466269 +0000 UTC m=+422.708598154"
Sep 17 17:41:06.355030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.354915    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-755946c6bf-hj2wl" podUID="26178025-53b6-489c-a608-c2a08889ab68" containerName="console" containerID="cri-o://206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce" gracePeriod=15
Sep 17 17:41:06.594584 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.594560    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:41:06.704672 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704603    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704672 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704647    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjqkt\" (UniqueName: \"kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704673    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704963 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704866    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704963 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704903    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.704963 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.704946    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca\") pod \"26178025-53b6-489c-a608-c2a08889ab68\" (UID: \"26178025-53b6-489c-a608-c2a08889ab68\") "
Sep 17 17:41:06.705136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config" (OuterVolumeSpecName: "console-config") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:06.705136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705043    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:06.705254 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705194    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-trusted-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.705254 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705212    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-console-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.705436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:06.705503 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.705480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca" (OuterVolumeSpecName: "service-ca") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:06.706825 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.706803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt" (OuterVolumeSpecName: "kube-api-access-wjqkt") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "kube-api-access-wjqkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:06.707365 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.707341    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:06.707445 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.707360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26178025-53b6-489c-a608-c2a08889ab68" (UID: "26178025-53b6-489c-a608-c2a08889ab68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:06.805898 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.805875    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-service-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.805978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.805909    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqkt\" (UniqueName: \"kubernetes.io/projected/26178025-53b6-489c-a608-c2a08889ab68-kube-api-access-wjqkt\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.805978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.805919    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.805978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.805928    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26178025-53b6-489c-a608-c2a08889ab68-console-oauth-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:06.805978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:06.805936    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26178025-53b6-489c-a608-c2a08889ab68-oauth-serving-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:41:07.383933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.382851    2575 generic.go:356] "Generic (PLEG): container finished" podID="26178025-53b6-489c-a608-c2a08889ab68" containerID="206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce" exitCode=0
Sep 17 17:41:07.383933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.382897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755946c6bf-hj2wl" event={"ID":"26178025-53b6-489c-a608-c2a08889ab68","Type":"ContainerDied","Data":"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"}
Sep 17 17:41:07.383933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.382945    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755946c6bf-hj2wl" event={"ID":"26178025-53b6-489c-a608-c2a08889ab68","Type":"ContainerDied","Data":"651d21a0a80f33b9b4ac0974f2418f4859a4243e7d603fd3632baa2bcf0e464c"}
Sep 17 17:41:07.383933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.382972    2575 scope.go:117] "RemoveContainer" containerID="206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"
Sep 17 17:41:07.383933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.383208    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755946c6bf-hj2wl"
Sep 17 17:41:07.393020 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.393001    2575 scope.go:117] "RemoveContainer" containerID="206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"
Sep 17 17:41:07.393451 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:41:07.393426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce\": container with ID starting with 206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce not found: ID does not exist" containerID="206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"
Sep 17 17:41:07.393548 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.393457    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce"} err="failed to get container status \"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce\": rpc error: code = NotFound desc = could not find container \"206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce\": container with ID starting with 206ab371514cf4a9dfd8961bb702e928d3a89e93c62b8586b43db30d3982b4ce not found: ID does not exist"
Sep 17 17:41:07.407434 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.407410    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:41:07.419359 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:07.419335    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-755946c6bf-hj2wl"]
Sep 17 17:41:08.776867 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:08.776833    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26178025-53b6-489c-a608-c2a08889ab68" path="/var/lib/kubelet/pods/26178025-53b6-489c-a608-c2a08889ab68/volumes"
Sep 17 17:41:22.332169 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:41:22.332131    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9b487cb79-cmxb5"
Sep 17 17:43:10.911006 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.910976    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911281    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26178025-53b6-489c-a608-c2a08889ab68" containerName="console"
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911292    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26178025-53b6-489c-a608-c2a08889ab68" containerName="console"
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911300    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" containerName="manager"
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911306    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" containerName="manager"
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911344    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26178025-53b6-489c-a608-c2a08889ab68" containerName="console"
Sep 17 17:43:10.911484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.911353    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d64dd4-d39f-4eac-89cb-a0eb0e6b21e6" containerName="manager"
Sep 17 17:43:10.914233 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.914210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:10.916868 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.916815    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:43:10.916868 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.916869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:43:10.917135 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.916909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:43:10.917135 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.917011    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:43:10.917859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.917835    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:43:10.917970 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.917894    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:43:10.917970 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.917928    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:43:10.924878 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:10.924855    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:43:11.016481 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.016448    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:43:11.019783 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.019761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.022352 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.022332    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-897d5-predictor-serving-cert\""
Sep 17 17:43:11.022595 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.022580    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-897d5-kube-rbac-proxy-sar-config\""
Sep 17 17:43:11.030583 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.030563    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:43:11.033281 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.033260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.033378 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.033305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522js\" (UniqueName: \"kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.033378 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.033333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.033463 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.033436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.033509 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.033483    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.134715 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.134820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.134820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.134820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.134820 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqj7\" (UniqueName: \"kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.135030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.135030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.135030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.135030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522js\" (UniqueName: \"kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.135030 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.134966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.135316 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.135036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.135316 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:43:11.135090    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:43:11.135316 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:43:11.135173    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls podName:4ffaea3e-67a7-4cbb-aaea-222d39974f4c nodeName:}" failed. No retries permitted until 2026-09-17 17:43:11.635153414 +0000 UTC m=+563.002285282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls") pod "isvc-canary-rollback-predictor-786544c6c8-7jbxc" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:43:11.135510 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.135491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.135560 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.135517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.147885 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.147865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-522js\" (UniqueName: \"kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.236225 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236225 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqj7\" (UniqueName: \"kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236357 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236357 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236357 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236602 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236581    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236863 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.236939 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.236904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.238481 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.238463    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.248393 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.248369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqj7\" (UniqueName: \"kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7\") pod \"isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.329397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.329379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:11.606909 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.606886    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:43:11.609142 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:43:11.609118    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b38bfd4_9af2_458c_942d_5fb86077a453.slice/crio-d8c8b90300b7bd721ff18f7004c454470f055a57d3fb67de2a1c494c26706663 WatchSource:0}: Error finding container d8c8b90300b7bd721ff18f7004c454470f055a57d3fb67de2a1c494c26706663: Status 404 returned error can't find the container with id d8c8b90300b7bd721ff18f7004c454470f055a57d3fb67de2a1c494c26706663
Sep 17 17:43:11.639886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.639863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.642019 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.641986    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-7jbxc\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:11.806515 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.806492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerStarted","Data":"d8c8b90300b7bd721ff18f7004c454470f055a57d3fb67de2a1c494c26706663"}
Sep 17 17:43:11.826053 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:11.826029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:12.097151 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:12.097119    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:43:12.100184 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:43:12.100159    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffaea3e_67a7_4cbb_aaea_222d39974f4c.slice/crio-57645e3e42b41278808b153c932cdb1f7ef674c9383018b422befa6b277d6004 WatchSource:0}: Error finding container 57645e3e42b41278808b153c932cdb1f7ef674c9383018b422befa6b277d6004: Status 404 returned error can't find the container with id 57645e3e42b41278808b153c932cdb1f7ef674c9383018b422befa6b277d6004
Sep 17 17:43:12.813091 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:12.813048    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerStarted","Data":"57645e3e42b41278808b153c932cdb1f7ef674c9383018b422befa6b277d6004"}
Sep 17 17:43:15.825463 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:15.825418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerStarted","Data":"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"}
Sep 17 17:43:15.827011 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:15.826987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerStarted","Data":"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"}
Sep 17 17:43:19.842852 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:19.842823    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerID="258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79" exitCode=0
Sep 17 17:43:19.843228 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:19.842897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerDied","Data":"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"}
Sep 17 17:43:19.844560 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:19.844541    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerID="10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6" exitCode=0
Sep 17 17:43:19.844638 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:19.844577    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerDied","Data":"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"}
Sep 17 17:43:35.946598 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:35.946485    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerStarted","Data":"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"}
Sep 17 17:43:35.953505 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:35.953449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerStarted","Data":"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"}
Sep 17 17:43:38.971185 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:38.971142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerStarted","Data":"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"}
Sep 17 17:43:38.973694 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:38.973663    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerStarted","Data":"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"}
Sep 17 17:43:38.973857 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:38.973837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:38.997972 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:38.997912    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podStartSLOduration=4.15600155 podStartE2EDuration="28.997884315s" totalImagesPullingTime="20.84259808s" totalInitContainerRuntime="3.999284685s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:38.995304906 +0000 UTC m=+590.362436786" watchObservedRunningTime="2026-09-17 17:43:38.997884315 +0000 UTC m=+590.365016197"
Sep 17 17:43:39.978518 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:39.978485    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:39.979714 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:39.979682    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:43:40.984591 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:40.984556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerStarted","Data":"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"}
Sep 17 17:43:40.984976 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:40.984952    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:43:41.009265 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.009227    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podStartSLOduration=3.5528883110000002 podStartE2EDuration="31.009213439s" totalImagesPullingTime="23.568330314s" totalInitContainerRuntime="3.887994814s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:10 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:41.00839024 +0000 UTC m=+592.375522120" watchObservedRunningTime="2026-09-17 17:43:41.009213439 +0000 UTC m=+592.376345320"
Sep 17 17:43:41.987882 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.987846    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:41.988271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.987921    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:41.988271 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.987956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:41.989022 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.988994    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:43:41.989650 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:41.989627    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:43:42.991878 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:42.991843    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:43:42.992388 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:42.992153    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:43:45.990270 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:45.990241    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:43:45.990714 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:45.990689    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:43:47.996060 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:47.996032    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:43:47.996629 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:47.996599    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:43:47.996895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:47.996862    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:43:50.713206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:50.713180    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:43:50.713608 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:50.713435    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:43:55.990964 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:55.990818    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:43:57.997286 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:57.997242    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:43:57.997727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:43:57.997704    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:05.990870 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:05.990780    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:07.996853 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:07.996804    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:44:07.997339 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:07.997312    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:15.991266 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:15.991212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:17.996524 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:17.996480    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:44:17.996986 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:17.996962    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:25.991220 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:25.991170    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:27.997112 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:27.997042    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:44:27.998594 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:27.997531    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:35.991278 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:35.991232    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:37.998758 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:37.998706    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:44:37.999606 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:37.999573    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:45.991170 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:45.991129    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:46.107073 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.107041    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:44:46.107381 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.107336    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" containerID="cri-o://44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747" gracePeriod=30
Sep 17 17:44:46.107517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.107448    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kube-rbac-proxy" containerID="cri-o://13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb" gracePeriod=30
Sep 17 17:44:46.107590 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.107525    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:44:46.243258 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.243203    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerID="13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb" exitCode=2
Sep 17 17:44:46.243351 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:46.243269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerDied","Data":"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"}
Sep 17 17:44:47.997276 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:47.997237    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:44:47.997678 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:47.997347    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:44:50.739936 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.739893    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:44:50.743347 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.743325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.758635 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.758615    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:44:50.762550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.762528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.762677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.762555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.762677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.762573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.762677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.762595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvst\" (UniqueName: \"kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.762677 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.762632    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.863625 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.863600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvst\" (UniqueName: \"kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.863776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.863631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.863776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.863682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.863776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.863698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.863776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.863712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.864197 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.864172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.864345 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.864329    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.864413 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.864400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.866048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.866027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.873896 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.873873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvst\" (UniqueName: \"kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst\") pod \"isvc-canary-rollback-predictor-786544c6c8-xd6mb\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:50.985992 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:50.985963    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:44:51.054601 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.054549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:51.327028 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.327006    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:44:51.329492 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:44:51.329443    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69ed496_aee7_4115_ae5a_a9460bcb2e88.slice/crio-c79518527c23a2c23fbb1ac1e36e589c96e6f60c0f4d964a9b5b9078ef97ef1c WatchSource:0}: Error finding container c79518527c23a2c23fbb1ac1e36e589c96e6f60c0f4d964a9b5b9078ef97ef1c: Status 404 returned error can't find the container with id c79518527c23a2c23fbb1ac1e36e589c96e6f60c0f4d964a9b5b9078ef97ef1c
Sep 17 17:44:51.745948 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.745918    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:44:51.771292 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771268    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert\") pod \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") "
Sep 17 17:44:51.771436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771316    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config\") pod \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") "
Sep 17 17:44:51.771436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771343    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location\") pod \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") "
Sep 17 17:44:51.771436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771378    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-522js\" (UniqueName: \"kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js\") pod \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") "
Sep 17 17:44:51.771648 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771466    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") pod \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\" (UID: \"4ffaea3e-67a7-4cbb-aaea-222d39974f4c\") "
Sep 17 17:44:51.771648 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ffaea3e-67a7-4cbb-aaea-222d39974f4c" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.771648 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771620    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ffaea3e-67a7-4cbb-aaea-222d39974f4c" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:51.771798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771683    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.771798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771700    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.771911 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.771859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ffaea3e-67a7-4cbb-aaea-222d39974f4c" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:51.774302 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.774277    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ffaea3e-67a7-4cbb-aaea-222d39974f4c" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:51.774426 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.774370    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js" (OuterVolumeSpecName: "kube-api-access-522js") pod "4ffaea3e-67a7-4cbb-aaea-222d39974f4c" (UID: "4ffaea3e-67a7-4cbb-aaea-222d39974f4c"). InnerVolumeSpecName "kube-api-access-522js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:51.872836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.872781    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.872836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.872811    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-522js\" (UniqueName: \"kubernetes.io/projected/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-kube-api-access-522js\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:44:51.872836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:51.872825    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ffaea3e-67a7-4cbb-aaea-222d39974f4c-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:44:52.267802 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.267696    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerID="44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747" exitCode=139
Sep 17 17:44:52.267802 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.267757    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"
Sep 17 17:44:52.268000 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.267793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerDied","Data":"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"}
Sep 17 17:44:52.268000 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.267876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc" event={"ID":"4ffaea3e-67a7-4cbb-aaea-222d39974f4c","Type":"ContainerDied","Data":"57645e3e42b41278808b153c932cdb1f7ef674c9383018b422befa6b277d6004"}
Sep 17 17:44:52.268000 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.267919    2575 scope.go:117] "RemoveContainer" containerID="13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"
Sep 17 17:44:52.269857 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.269826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerStarted","Data":"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"}
Sep 17 17:44:52.269962 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.269900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerStarted","Data":"c79518527c23a2c23fbb1ac1e36e589c96e6f60c0f4d964a9b5b9078ef97ef1c"}
Sep 17 17:44:52.277356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.277337    2575 scope.go:117] "RemoveContainer" containerID="44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"
Sep 17 17:44:52.285062 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.285042    2575 scope.go:117] "RemoveContainer" containerID="10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"
Sep 17 17:44:52.292300 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.292282    2575 scope.go:117] "RemoveContainer" containerID="13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"
Sep 17 17:44:52.292604 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:44:52.292584    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb\": container with ID starting with 13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb not found: ID does not exist" containerID="13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"
Sep 17 17:44:52.292647 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.292612    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb"} err="failed to get container status \"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb\": rpc error: code = NotFound desc = could not find container \"13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb\": container with ID starting with 13f339700438cebb778cebbf0ff13baef47d659c20ea7f337a01c34325fdbfeb not found: ID does not exist"
Sep 17 17:44:52.292647 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.292630    2575 scope.go:117] "RemoveContainer" containerID="44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"
Sep 17 17:44:52.292918 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:44:52.292898    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747\": container with ID starting with 44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747 not found: ID does not exist" containerID="44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"
Sep 17 17:44:52.292956 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.292924    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747"} err="failed to get container status \"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747\": rpc error: code = NotFound desc = could not find container \"44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747\": container with ID starting with 44eed4b7b70341885237464de229433e6aba359930593aa28e01707963e63747 not found: ID does not exist"
Sep 17 17:44:52.292956 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.292941    2575 scope.go:117] "RemoveContainer" containerID="10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"
Sep 17 17:44:52.293241 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:44:52.293220    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6\": container with ID starting with 10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6 not found: ID does not exist" containerID="10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"
Sep 17 17:44:52.293295 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.293247    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6"} err="failed to get container status \"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6\": rpc error: code = NotFound desc = could not find container \"10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6\": container with ID starting with 10e5a181c0daa34b450268692488241961c17c32aeb114349eff3a032fa320e6 not found: ID does not exist"
Sep 17 17:44:52.311716 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.311685    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:44:52.313526 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.313505    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-7jbxc"]
Sep 17 17:44:52.777680 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:52.777655    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" path="/var/lib/kubelet/pods/4ffaea3e-67a7-4cbb-aaea-222d39974f4c/volumes"
Sep 17 17:44:55.285360 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.285331    2575 generic.go:356] "Generic (PLEG): container finished" podID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerID="cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858" exitCode=0
Sep 17 17:44:55.285721 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.285392    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerDied","Data":"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"}
Sep 17 17:44:55.708477 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.708440    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:44:55.708783 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.708756    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" containerID="cri-o://afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857" gracePeriod=30
Sep 17 17:44:55.708865 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.708765    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" containerID="cri-o://bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea" gracePeriod=30
Sep 17 17:44:55.708926 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:55.708771    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" containerID="cri-o://a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22" gracePeriod=30
Sep 17 17:44:56.042873 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.042794    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:44:56.043176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043156    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="storage-initializer"
Sep 17 17:44:56.043176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043178    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="storage-initializer"
Sep 17 17:44:56.043321 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043204    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container"
Sep 17 17:44:56.043321 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043213    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container"
Sep 17 17:44:56.043321 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043229    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kube-rbac-proxy"
Sep 17 17:44:56.043321 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043238    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kube-rbac-proxy"
Sep 17 17:44:56.043522 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043347    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kube-rbac-proxy"
Sep 17 17:44:56.043522 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.043358    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffaea3e-67a7-4cbb-aaea-222d39974f4c" containerName="kserve-container"
Sep 17 17:44:56.045523 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.045504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.048074 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.048051    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-636c4-kube-rbac-proxy-sar-config\""
Sep 17 17:44:56.048074 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.048064    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-636c4-predictor-serving-cert\""
Sep 17 17:44:56.058306 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.058285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:44:56.100671 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.100649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.100762 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.100677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.100762 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.100698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpc2\" (UniqueName: \"kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.100844 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.100809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.100844 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.100829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201156 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201369 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201413 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201455 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc2\" (UniqueName: \"kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.201649 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.201625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.202019 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.202002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.202130 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.202091    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.203461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.203441    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.214889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.214871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpc2\" (UniqueName: \"kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2\") pod \"isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.291765 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.291738    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerID="a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22" exitCode=2
Sep 17 17:44:56.292056 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.291808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerDied","Data":"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"}
Sep 17 17:44:56.293903 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.293857    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerStarted","Data":"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"}
Sep 17 17:44:56.293972 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.293905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerStarted","Data":"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"}
Sep 17 17:44:56.294164 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.294140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:56.314758 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.314713    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podStartSLOduration=2.48034963 podStartE2EDuration="6.314700525s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.834350895s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:56.313536492 +0000 UTC m=+667.680668374" watchObservedRunningTime="2026-09-17 17:44:56.314700525 +0000 UTC m=+667.681832406"
Sep 17 17:44:56.355775 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.355754    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:44:56.625350 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:56.625325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:44:56.627217 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:44:56.627185    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38329cad_31ab_49e1_87a2_0ed33be53588.slice/crio-bbe69bf7b5f877f7f3253e3ebffe7df402be4aa20d76faf7571ccce00060e8d4 WatchSource:0}: Error finding container bbe69bf7b5f877f7f3253e3ebffe7df402be4aa20d76faf7571ccce00060e8d4: Status 404 returned error can't find the container with id bbe69bf7b5f877f7f3253e3ebffe7df402be4aa20d76faf7571ccce00060e8d4
Sep 17 17:44:57.300136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.300081    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerStarted","Data":"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"}
Sep 17 17:44:57.300586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.300171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerStarted","Data":"bbe69bf7b5f877f7f3253e3ebffe7df402be4aa20d76faf7571ccce00060e8d4"}
Sep 17 17:44:57.300586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.300546    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:44:57.302056 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.302030    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:44:57.992770 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.992732    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:44:57.997018 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.996994    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:44:57.997453 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:57.997428    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:44:58.304775 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:44:58.304681    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:00.314652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:00.314625    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerID="afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857" exitCode=0
Sep 17 17:45:00.314969 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:00.314664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerDied","Data":"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"}
Sep 17 17:45:01.319944 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:01.319911    2575 generic.go:356] "Generic (PLEG): container finished" podID="38329cad-31ab-49e1-87a2-0ed33be53588" containerID="4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85" exitCode=0
Sep 17 17:45:01.320330 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:01.319989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerDied","Data":"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"}
Sep 17 17:45:02.992907 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:02.992866    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:45:03.310969 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:03.310589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:45:03.311560 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:03.311522    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:07.992085 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:07.992041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:45:07.992518 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:07.992187    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:45:07.997144 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:07.997078    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:45:07.997440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:07.997412    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:45:12.992378 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:12.992336    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:45:13.311269 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:13.311188    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:17.992997 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:17.992957    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:45:17.997424 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:17.997398    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:45:17.997550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:17.997528    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:45:17.997771 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:17.997745    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:45:17.997851 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:17.997842    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:45:20.404936 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:20.404895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerStarted","Data":"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"}
Sep 17 17:45:20.405392 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:20.404986    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerStarted","Data":"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"}
Sep 17 17:45:20.405392 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:20.405248    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:45:20.425372 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:20.425326    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podStartSLOduration=2.538193797 podStartE2EDuration="24.425312647s" totalImagesPullingTime="18.138124515s" totalInitContainerRuntime="3.748994335s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:20.424041057 +0000 UTC m=+691.791172938" watchObservedRunningTime="2026-09-17 17:45:20.425312647 +0000 UTC m=+691.792444527"
Sep 17 17:45:21.408706 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:21.408676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:45:21.410184 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:21.410157    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:45:22.412055 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:22.412012    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:45:22.992206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:22.992150    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:45:23.311388 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:23.311311    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:26.422614 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.422589    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:45:26.429460 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.429431    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerID="bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea" exitCode=0
Sep 17 17:45:26.429570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.429489    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"
Sep 17 17:45:26.429570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.429493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerDied","Data":"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"}
Sep 17 17:45:26.429570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.429559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7" event={"ID":"7b38bfd4-9af2-458c-942d-5fb86077a453","Type":"ContainerDied","Data":"d8c8b90300b7bd721ff18f7004c454470f055a57d3fb67de2a1c494c26706663"}
Sep 17 17:45:26.429749 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.429590    2575 scope.go:117] "RemoveContainer" containerID="bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"
Sep 17 17:45:26.437063 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.437048    2575 scope.go:117] "RemoveContainer" containerID="a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"
Sep 17 17:45:26.444019 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.444004    2575 scope.go:117] "RemoveContainer" containerID="afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"
Sep 17 17:45:26.451446 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.451427    2575 scope.go:117] "RemoveContainer" containerID="258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"
Sep 17 17:45:26.458679 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.458660    2575 scope.go:117] "RemoveContainer" containerID="bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"
Sep 17 17:45:26.458952 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:45:26.458933    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea\": container with ID starting with bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea not found: ID does not exist" containerID="bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"
Sep 17 17:45:26.459026 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.458966    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea"} err="failed to get container status \"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea\": rpc error: code = NotFound desc = could not find container \"bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea\": container with ID starting with bc40e44de5e482d85be43159aef530b907e52ac449e949439b62ecfcdf258cea not found: ID does not exist"
Sep 17 17:45:26.459026 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.458991    2575 scope.go:117] "RemoveContainer" containerID="a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"
Sep 17 17:45:26.459278 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:45:26.459259    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22\": container with ID starting with a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22 not found: ID does not exist" containerID="a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"
Sep 17 17:45:26.459391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.459284    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22"} err="failed to get container status \"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22\": rpc error: code = NotFound desc = could not find container \"a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22\": container with ID starting with a11bb39ce9441354cecd085797417c9555b0710214034d2c119fcaa686d0fe22 not found: ID does not exist"
Sep 17 17:45:26.459391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.459304    2575 scope.go:117] "RemoveContainer" containerID="afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"
Sep 17 17:45:26.459536 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:45:26.459518    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857\": container with ID starting with afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857 not found: ID does not exist" containerID="afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"
Sep 17 17:45:26.459577 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.459543    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857"} err="failed to get container status \"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857\": rpc error: code = NotFound desc = could not find container \"afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857\": container with ID starting with afced3e3a88dd8c4376371e204941004c8b12040aa854c4d9bc2a34f3e001857 not found: ID does not exist"
Sep 17 17:45:26.459577 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.459558    2575 scope.go:117] "RemoveContainer" containerID="258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"
Sep 17 17:45:26.459830 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:45:26.459815    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79\": container with ID starting with 258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79 not found: ID does not exist" containerID="258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"
Sep 17 17:45:26.459877 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.459834    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79"} err="failed to get container status \"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79\": rpc error: code = NotFound desc = could not find container \"258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79\": container with ID starting with 258e3253488282feb5d300afa25d6aadcd0de61787668c9ab9464094a95e7a79 not found: ID does not exist"
Sep 17 17:45:26.543256 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config\") pod \"7b38bfd4-9af2-458c-942d-5fb86077a453\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") "
Sep 17 17:45:26.543361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543263    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sqj7\" (UniqueName: \"kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7\") pod \"7b38bfd4-9af2-458c-942d-5fb86077a453\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") "
Sep 17 17:45:26.543361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543290    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert\") pod \"7b38bfd4-9af2-458c-942d-5fb86077a453\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") "
Sep 17 17:45:26.543361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls\") pod \"7b38bfd4-9af2-458c-942d-5fb86077a453\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") "
Sep 17 17:45:26.543361 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543337    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location\") pod \"7b38bfd4-9af2-458c-942d-5fb86077a453\" (UID: \"7b38bfd4-9af2-458c-942d-5fb86077a453\") "
Sep 17 17:45:26.543718 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543597    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b38bfd4-9af2-458c-942d-5fb86077a453" (UID: "7b38bfd4-9af2-458c-942d-5fb86077a453"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:45:26.543813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543713    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b38bfd4-9af2-458c-942d-5fb86077a453" (UID: "7b38bfd4-9af2-458c-942d-5fb86077a453"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:45:26.543813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.543749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b38bfd4-9af2-458c-942d-5fb86077a453" (UID: "7b38bfd4-9af2-458c-942d-5fb86077a453"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:26.545362 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.545339    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b38bfd4-9af2-458c-942d-5fb86077a453" (UID: "7b38bfd4-9af2-458c-942d-5fb86077a453"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:26.545484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.545464    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7" (OuterVolumeSpecName: "kube-api-access-6sqj7") pod "7b38bfd4-9af2-458c-942d-5fb86077a453" (UID: "7b38bfd4-9af2-458c-942d-5fb86077a453"). InnerVolumeSpecName "kube-api-access-6sqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:26.644131 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.644105    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b38bfd4-9af2-458c-942d-5fb86077a453-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:45:26.644131 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.644128    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:45:26.644270 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.644138    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sqj7\" (UniqueName: \"kubernetes.io/projected/7b38bfd4-9af2-458c-942d-5fb86077a453-kube-api-access-6sqj7\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:45:26.644270 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.644147    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b38bfd4-9af2-458c-942d-5fb86077a453-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:45:26.644270 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.644156    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b38bfd4-9af2-458c-942d-5fb86077a453-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:45:26.783060 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.783037    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:45:26.791448 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:26.791426    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-897d5-predictor-57d44f6c79-7wkm7"]
Sep 17 17:45:27.416895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:27.416865    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:45:27.417391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:27.417366    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:45:28.776397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:28.776364    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" path="/var/lib/kubelet/pods/7b38bfd4-9af2-458c-942d-5fb86077a453/volumes"
Sep 17 17:45:33.311664 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:33.311580    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:37.417664 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:37.417621    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:45:43.311297 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:43.311258    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:47.417660 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:47.417618    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:45:53.311661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:53.311621    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:45:57.417340 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:45:57.417300    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:46:03.312059 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:03.312018    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:46:07.418291 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:07.418257    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:46:13.312296 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:13.312258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:46:17.417591 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:17.417549    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:46:20.778012 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:20.777981    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:46:20.778536 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:20.778209    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" containerID="cri-o://b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9" gracePeriod=30
Sep 17 17:46:20.778536 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:20.778262    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kube-rbac-proxy" containerID="cri-o://da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b" gracePeriod=30
Sep 17 17:46:21.634192 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:21.634153    2575 generic.go:356] "Generic (PLEG): container finished" podID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerID="da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b" exitCode=2
Sep 17 17:46:21.634369 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:21.634227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerDied","Data":"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"}
Sep 17 17:46:23.305580 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:23.305540    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:46:23.311108 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:23.311070    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:46:24.604652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.604631    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:46:24.642011 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.641984    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kvst\" (UniqueName: \"kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst\") pod \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") "
Sep 17 17:46:24.642143 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642029    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location\") pod \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") "
Sep 17 17:46:24.642143 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert\") pod \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") "
Sep 17 17:46:24.642143 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls\") pod \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") "
Sep 17 17:46:24.642313 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642177    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config\") pod \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\" (UID: \"e69ed496-aee7-4115-ae5a-a9460bcb2e88\") "
Sep 17 17:46:24.642389 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642362    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e69ed496-aee7-4115-ae5a-a9460bcb2e88" (UID: "e69ed496-aee7-4115-ae5a-a9460bcb2e88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:24.642509 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642464    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e69ed496-aee7-4115-ae5a-a9460bcb2e88" (UID: "e69ed496-aee7-4115-ae5a-a9460bcb2e88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:24.642636 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.642568    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e69ed496-aee7-4115-ae5a-a9460bcb2e88" (UID: "e69ed496-aee7-4115-ae5a-a9460bcb2e88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:24.644274 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.644234    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e69ed496-aee7-4115-ae5a-a9460bcb2e88" (UID: "e69ed496-aee7-4115-ae5a-a9460bcb2e88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:24.644376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.644322    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst" (OuterVolumeSpecName: "kube-api-access-9kvst") pod "e69ed496-aee7-4115-ae5a-a9460bcb2e88" (UID: "e69ed496-aee7-4115-ae5a-a9460bcb2e88"). InnerVolumeSpecName "kube-api-access-9kvst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:24.647009 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.646984    2575 generic.go:356] "Generic (PLEG): container finished" podID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerID="b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9" exitCode=0
Sep 17 17:46:24.647118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.647023    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerDied","Data":"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"}
Sep 17 17:46:24.647118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.647031    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"
Sep 17 17:46:24.647118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.647076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb" event={"ID":"e69ed496-aee7-4115-ae5a-a9460bcb2e88","Type":"ContainerDied","Data":"c79518527c23a2c23fbb1ac1e36e589c96e6f60c0f4d964a9b5b9078ef97ef1c"}
Sep 17 17:46:24.647118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.647120    2575 scope.go:117] "RemoveContainer" containerID="da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"
Sep 17 17:46:24.656511 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.656496    2575 scope.go:117] "RemoveContainer" containerID="b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"
Sep 17 17:46:24.663060 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.663042    2575 scope.go:117] "RemoveContainer" containerID="cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"
Sep 17 17:46:24.669567 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.669550    2575 scope.go:117] "RemoveContainer" containerID="da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"
Sep 17 17:46:24.669821 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:24.669804    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b\": container with ID starting with da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b not found: ID does not exist" containerID="da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"
Sep 17 17:46:24.669871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.669829    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b"} err="failed to get container status \"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b\": rpc error: code = NotFound desc = could not find container \"da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b\": container with ID starting with da009fb8be93c8242c95cee780195ff21ea9fec14c3805ba507fcfa999a6665b not found: ID does not exist"
Sep 17 17:46:24.669871 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.669845    2575 scope.go:117] "RemoveContainer" containerID="b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"
Sep 17 17:46:24.670205 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:24.670078    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9\": container with ID starting with b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9 not found: ID does not exist" containerID="b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"
Sep 17 17:46:24.670255 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.670215    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9"} err="failed to get container status \"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9\": rpc error: code = NotFound desc = could not find container \"b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9\": container with ID starting with b6e9690b4f2fd50019e6c3835fe0193e146e4a86e86e72765eec2a2aad5757c9 not found: ID does not exist"
Sep 17 17:46:24.670255 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.670234    2575 scope.go:117] "RemoveContainer" containerID="cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"
Sep 17 17:46:24.670654 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:24.670627    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858\": container with ID starting with cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858 not found: ID does not exist" containerID="cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"
Sep 17 17:46:24.670901 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.670660    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858"} err="failed to get container status \"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858\": rpc error: code = NotFound desc = could not find container \"cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858\": container with ID starting with cb6ee2c24f4de84e973277c95cb5f7dc05cc612314e40093c60d6896cdaa3858 not found: ID does not exist"
Sep 17 17:46:24.672618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.672593    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:46:24.680673 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.680652    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-xd6mb"]
Sep 17 17:46:24.742973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.742926    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvst\" (UniqueName: \"kubernetes.io/projected/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-api-access-9kvst\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:24.742973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.742945    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:24.742973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.742956    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:24.742973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.742964    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e69ed496-aee7-4115-ae5a-a9460bcb2e88-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:24.742973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.742972    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e69ed496-aee7-4115-ae5a-a9460bcb2e88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:24.777025 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:24.776998    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" path="/var/lib/kubelet/pods/e69ed496-aee7-4115-ae5a-a9460bcb2e88/volumes"
Sep 17 17:46:26.002530 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002497    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002751    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="storage-initializer"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002764    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="storage-initializer"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002777    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002783    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002791    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kube-rbac-proxy"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002796    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kube-rbac-proxy"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002804    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="storage-initializer"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002809    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="storage-initializer"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002819    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002824    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002831    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002836    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002843    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy"
Sep 17 17:46:26.002879 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002848    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy"
Sep 17 17:46:26.003308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002891    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kserve-container"
Sep 17 17:46:26.003308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002897    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="kube-rbac-proxy"
Sep 17 17:46:26.003308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002903    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b38bfd4-9af2-458c-942d-5fb86077a453" containerName="agent"
Sep 17 17:46:26.003308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002911    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kube-rbac-proxy"
Sep 17 17:46:26.003308 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.002916    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69ed496-aee7-4115-ae5a-a9460bcb2e88" containerName="kserve-container"
Sep 17 17:46:26.005802 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.005786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.008124 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.008086    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 17:46:26.008224 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.008150    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:46:26.018244 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.018222    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:46:26.046945 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.046917    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:46:26.049289 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.049270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.051509 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.051623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.051623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051565    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:46:26.051623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.051623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.051826 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.051665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqrs\" (UniqueName: \"kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.060357 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.060333    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:46:26.152925 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.152891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.152927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.153061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.152961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.152984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.153061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tgt\" (UniqueName: \"kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.153061 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqrs\" (UniqueName: \"kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.153385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.153514 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153625 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.153826 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.153810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.155665 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.155645    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.162388 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.162365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqrs\" (UniqueName: \"kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.253792 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.253739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.253792 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.253773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.253955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.253792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tgt\" (UniqueName: \"kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.253955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.253846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.253955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.253877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.254310 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.254291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.254574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.254544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.254690 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.254544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.256028 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.256011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.269893 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.269872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tgt\" (UniqueName: \"kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt\") pod \"isvc-canary-stop-predictor-684df5466b-njrqf\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.316039 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.316016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:26.358979 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.358950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:26.586796 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.586772    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:46:26.588548 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:46:26.588520    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29667b89_38ad_4535_8b59_532289ebf1af.slice/crio-c57f7a127a41070634242b86f4dfeac6fd1dab5ee70a725bebfe0cfe9bf1987d WatchSource:0}: Error finding container c57f7a127a41070634242b86f4dfeac6fd1dab5ee70a725bebfe0cfe9bf1987d: Status 404 returned error can't find the container with id c57f7a127a41070634242b86f4dfeac6fd1dab5ee70a725bebfe0cfe9bf1987d
Sep 17 17:46:26.636206 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.636178    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:46:26.638752 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:46:26.638726    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147f60e0_4a29_4a41_938c_f760898b17ce.slice/crio-b9cd66d58b92939068107d62036707819d318aeae32d9da37eb914458fe5b9cf WatchSource:0}: Error finding container b9cd66d58b92939068107d62036707819d318aeae32d9da37eb914458fe5b9cf: Status 404 returned error can't find the container with id b9cd66d58b92939068107d62036707819d318aeae32d9da37eb914458fe5b9cf
Sep 17 17:46:26.656943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.656916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerStarted","Data":"b9cd66d58b92939068107d62036707819d318aeae32d9da37eb914458fe5b9cf"}
Sep 17 17:46:26.658423 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.658395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerStarted","Data":"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"}
Sep 17 17:46:26.658525 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:26.658458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerStarted","Data":"c57f7a127a41070634242b86f4dfeac6fd1dab5ee70a725bebfe0cfe9bf1987d"}
Sep 17 17:46:27.418317 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:27.418282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:46:27.663542 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:27.663497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerStarted","Data":"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"}
Sep 17 17:46:30.675908 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:30.675880    2575 generic.go:356] "Generic (PLEG): container finished" podID="147f60e0-4a29-4a41-938c-f760898b17ce" containerID="e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460" exitCode=0
Sep 17 17:46:30.676221 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:30.675953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerDied","Data":"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"}
Sep 17 17:46:30.677525 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:30.677507    2575 generic.go:356] "Generic (PLEG): container finished" podID="29667b89-38ad-4535-8b59-532289ebf1af" containerID="fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b" exitCode=0
Sep 17 17:46:30.677602 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:30.677559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerDied","Data":"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"}
Sep 17 17:46:31.683569 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.683538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerStarted","Data":"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"}
Sep 17 17:46:31.683959 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.683603    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerStarted","Data":"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"}
Sep 17 17:46:31.683959 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.683818    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:31.685556 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.685535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerStarted","Data":"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"}
Sep 17 17:46:31.685645 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.685576    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerStarted","Data":"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"}
Sep 17 17:46:31.685764 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.685746    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:31.685809 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.685799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:31.686973 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.686949    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:46:31.704953 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.704919    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podStartSLOduration=1.871041696 podStartE2EDuration="5.704908132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833866436s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:31.70469698 +0000 UTC m=+763.071828861" watchObservedRunningTime="2026-09-17 17:46:31.704908132 +0000 UTC m=+763.072040013"
Sep 17 17:46:31.730164 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:31.726789    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podStartSLOduration=2.87490989 podStartE2EDuration="6.726772606s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.851862716s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:31.725484692 +0000 UTC m=+763.092616587" watchObservedRunningTime="2026-09-17 17:46:31.726772606 +0000 UTC m=+763.093904489"
Sep 17 17:46:32.689900 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:32.689849    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:46:32.690407 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:32.690093    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:32.691022 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:32.690997    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:46:33.693951 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:33.693914    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:46:37.694172 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:37.694141    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:46:37.694599 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:37.694574    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:46:38.698857 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:38.698825    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:46:38.699348 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:38.699322    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:46:46.078517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:46.078484    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:46:46.079069 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:46.078825    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" containerID="cri-o://8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8" gracePeriod=30
Sep 17 17:46:46.079069 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:46.078892    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kube-rbac-proxy" containerID="cri-o://9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748" gracePeriod=30
Sep 17 17:46:46.743940 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:46.743909    2575 generic.go:356] "Generic (PLEG): container finished" podID="38329cad-31ab-49e1-87a2-0ed33be53588" containerID="9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748" exitCode=2
Sep 17 17:46:46.744111 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:46.743981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerDied","Data":"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"}
Sep 17 17:46:47.412543 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:47.412507    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 17:46:47.417888 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:47.417863    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:46:47.695118 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:47.695011    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:46:48.699618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:48.699579    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:46:50.316783 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.316761    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:46:50.425550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425491    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config\") pod \"38329cad-31ab-49e1-87a2-0ed33be53588\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") "
Sep 17 17:46:50.425550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425522    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert\") pod \"38329cad-31ab-49e1-87a2-0ed33be53588\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") "
Sep 17 17:46:50.425721 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425555    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls\") pod \"38329cad-31ab-49e1-87a2-0ed33be53588\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") "
Sep 17 17:46:50.425721 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425597    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location\") pod \"38329cad-31ab-49e1-87a2-0ed33be53588\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") "
Sep 17 17:46:50.425721 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxpc2\" (UniqueName: \"kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2\") pod \"38329cad-31ab-49e1-87a2-0ed33be53588\" (UID: \"38329cad-31ab-49e1-87a2-0ed33be53588\") "
Sep 17 17:46:50.425931 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425907    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38329cad-31ab-49e1-87a2-0ed33be53588" (UID: "38329cad-31ab-49e1-87a2-0ed33be53588"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:50.425995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425931    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38329cad-31ab-49e1-87a2-0ed33be53588" (UID: "38329cad-31ab-49e1-87a2-0ed33be53588"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:50.425995 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.425959    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38329cad-31ab-49e1-87a2-0ed33be53588" (UID: "38329cad-31ab-49e1-87a2-0ed33be53588"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:50.427750 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.427717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38329cad-31ab-49e1-87a2-0ed33be53588" (UID: "38329cad-31ab-49e1-87a2-0ed33be53588"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:50.427862 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.427777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2" (OuterVolumeSpecName: "kube-api-access-nxpc2") pod "38329cad-31ab-49e1-87a2-0ed33be53588" (UID: "38329cad-31ab-49e1-87a2-0ed33be53588"). InnerVolumeSpecName "kube-api-access-nxpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:50.526148 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.526092    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpc2\" (UniqueName: \"kubernetes.io/projected/38329cad-31ab-49e1-87a2-0ed33be53588-kube-api-access-nxpc2\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.526148 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.526146    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.526287 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.526167    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38329cad-31ab-49e1-87a2-0ed33be53588-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.526287 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.526176    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38329cad-31ab-49e1-87a2-0ed33be53588-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.526287 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.526184    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38329cad-31ab-49e1-87a2-0ed33be53588-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.760696 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.760664    2575 generic.go:356] "Generic (PLEG): container finished" podID="38329cad-31ab-49e1-87a2-0ed33be53588" containerID="8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8" exitCode=0
Sep 17 17:46:50.760798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.760722    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"
Sep 17 17:46:50.760798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.760747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerDied","Data":"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"}
Sep 17 17:46:50.760891 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.760815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6" event={"ID":"38329cad-31ab-49e1-87a2-0ed33be53588","Type":"ContainerDied","Data":"bbe69bf7b5f877f7f3253e3ebffe7df402be4aa20d76faf7571ccce00060e8d4"}
Sep 17 17:46:50.760891 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.760855    2575 scope.go:117] "RemoveContainer" containerID="9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"
Sep 17 17:46:50.768886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.768873    2575 scope.go:117] "RemoveContainer" containerID="8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"
Sep 17 17:46:50.781346 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.781326    2575 scope.go:117] "RemoveContainer" containerID="4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"
Sep 17 17:46:50.785235 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.785211    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:46:50.789119 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789084    2575 scope.go:117] "RemoveContainer" containerID="9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"
Sep 17 17:46:50.789302 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789284    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-636c4-predictor-754545887c-2gcn6"]
Sep 17 17:46:50.789386 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:50.789370    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748\": container with ID starting with 9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748 not found: ID does not exist" containerID="9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"
Sep 17 17:46:50.789448 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789394    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748"} err="failed to get container status \"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748\": rpc error: code = NotFound desc = could not find container \"9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748\": container with ID starting with 9e6e096f90e7e357678c73b93d7fbb92f0e9825a27b755fe76520eadc57e6748 not found: ID does not exist"
Sep 17 17:46:50.789448 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789409    2575 scope.go:117] "RemoveContainer" containerID="8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"
Sep 17 17:46:50.789674 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:50.789658    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8\": container with ID starting with 8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8 not found: ID does not exist" containerID="8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"
Sep 17 17:46:50.789730 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789679    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8"} err="failed to get container status \"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8\": rpc error: code = NotFound desc = could not find container \"8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8\": container with ID starting with 8c46fe60054c597fa1d39ab04497de86124b7654e56ba9b1b62e2c90f8d8a9b8 not found: ID does not exist"
Sep 17 17:46:50.789730 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789694    2575 scope.go:117] "RemoveContainer" containerID="4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"
Sep 17 17:46:50.789930 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:46:50.789912    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85\": container with ID starting with 4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85 not found: ID does not exist" containerID="4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"
Sep 17 17:46:50.789980 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:50.789936    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85"} err="failed to get container status \"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85\": rpc error: code = NotFound desc = could not find container \"4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85\": container with ID starting with 4d10ddd81472b50bca27133f7ef28416fa48085dd38e6be855029aeba2864d85 not found: ID does not exist"
Sep 17 17:46:52.776616 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:52.776580    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" path="/var/lib/kubelet/pods/38329cad-31ab-49e1-87a2-0ed33be53588/volumes"
Sep 17 17:46:57.694576 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:57.694529    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:46:58.700286 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:46:58.700244    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:47:07.695324 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:07.695231    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:47:08.700046 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:08.700010    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:47:17.694858 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:17.694819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:47:18.699306 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:18.699263    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:47:27.694854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:27.694808    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:47:28.700265 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:28.700222    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:47:37.694751 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:37.694712    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:47:38.699834 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:38.699796    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:47:47.695994 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.695962    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:47:47.821799 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.821763    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:47:47.822441 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.822386    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container" containerID="cri-o://76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0" gracePeriod=30
Sep 17 17:47:47.823832 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.823509    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kube-rbac-proxy" containerID="cri-o://799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f" gracePeriod=30
Sep 17 17:47:47.825163 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.825138    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:47:47.966655 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.966584    2575 generic.go:356] "Generic (PLEG): container finished" podID="147f60e0-4a29-4a41-938c-f760898b17ce" containerID="799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f" exitCode=2
Sep 17 17:47:47.966784 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:47.966655    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerDied","Data":"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"}
Sep 17 17:47:48.694644 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:48.694606    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 17:47:51.350290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.350267    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:47:51.360489 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls\") pod \"147f60e0-4a29-4a41-938c-f760898b17ce\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") "
Sep 17 17:47:51.360568 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360516    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location\") pod \"147f60e0-4a29-4a41-938c-f760898b17ce\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") "
Sep 17 17:47:51.360568 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360543    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config\") pod \"147f60e0-4a29-4a41-938c-f760898b17ce\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") "
Sep 17 17:47:51.360568 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert\") pod \"147f60e0-4a29-4a41-938c-f760898b17ce\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") "
Sep 17 17:47:51.360680 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360587    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4tgt\" (UniqueName: \"kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt\") pod \"147f60e0-4a29-4a41-938c-f760898b17ce\" (UID: \"147f60e0-4a29-4a41-938c-f760898b17ce\") "
Sep 17 17:47:51.360892 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "147f60e0-4a29-4a41-938c-f760898b17ce" (UID: "147f60e0-4a29-4a41-938c-f760898b17ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:51.360988 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360926    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "147f60e0-4a29-4a41-938c-f760898b17ce" (UID: "147f60e0-4a29-4a41-938c-f760898b17ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:51.360988 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.360930    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "147f60e0-4a29-4a41-938c-f760898b17ce" (UID: "147f60e0-4a29-4a41-938c-f760898b17ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:51.362454 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.362436    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "147f60e0-4a29-4a41-938c-f760898b17ce" (UID: "147f60e0-4a29-4a41-938c-f760898b17ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:51.362586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.362563    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt" (OuterVolumeSpecName: "kube-api-access-l4tgt") pod "147f60e0-4a29-4a41-938c-f760898b17ce" (UID: "147f60e0-4a29-4a41-938c-f760898b17ce"). InnerVolumeSpecName "kube-api-access-l4tgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:51.462008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.461948    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147f60e0-4a29-4a41-938c-f760898b17ce-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:51.462008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.461973    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:51.462008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.461982    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147f60e0-4a29-4a41-938c-f760898b17ce-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:51.462008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.461991    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4tgt\" (UniqueName: \"kubernetes.io/projected/147f60e0-4a29-4a41-938c-f760898b17ce-kube-api-access-l4tgt\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:51.462008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.462000    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147f60e0-4a29-4a41-938c-f760898b17ce-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:51.981252 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.981222    2575 generic.go:356] "Generic (PLEG): container finished" podID="147f60e0-4a29-4a41-938c-f760898b17ce" containerID="76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0" exitCode=0
Sep 17 17:47:51.981391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.981292    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"
Sep 17 17:47:51.981391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.981308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerDied","Data":"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"}
Sep 17 17:47:51.981391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.981377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf" event={"ID":"147f60e0-4a29-4a41-938c-f760898b17ce","Type":"ContainerDied","Data":"b9cd66d58b92939068107d62036707819d318aeae32d9da37eb914458fe5b9cf"}
Sep 17 17:47:51.981509 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.981417    2575 scope.go:117] "RemoveContainer" containerID="799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"
Sep 17 17:47:51.989618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.989599    2575 scope.go:117] "RemoveContainer" containerID="76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"
Sep 17 17:47:51.997011 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:51.996995    2575 scope.go:117] "RemoveContainer" containerID="e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"
Sep 17 17:47:52.002067 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.002047    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:47:52.004156 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.004136    2575 scope.go:117] "RemoveContainer" containerID="799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"
Sep 17 17:47:52.004415 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:47:52.004399    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f\": container with ID starting with 799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f not found: ID does not exist" containerID="799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"
Sep 17 17:47:52.004482 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.004422    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f"} err="failed to get container status \"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f\": rpc error: code = NotFound desc = could not find container \"799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f\": container with ID starting with 799b88f8be454ca91c27ee270e1f99f32a6b2e561bd4929e4e9d896d017f488f not found: ID does not exist"
Sep 17 17:47:52.004482 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.004438    2575 scope.go:117] "RemoveContainer" containerID="76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"
Sep 17 17:47:52.004702 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:47:52.004679    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0\": container with ID starting with 76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0 not found: ID does not exist" containerID="76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"
Sep 17 17:47:52.004787 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.004713    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0"} err="failed to get container status \"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0\": rpc error: code = NotFound desc = could not find container \"76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0\": container with ID starting with 76faff2419bce5b92a608450f46c0124130c10d1f709b0f2e658fee2c08ff1c0 not found: ID does not exist"
Sep 17 17:47:52.004787 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.004736    2575 scope.go:117] "RemoveContainer" containerID="e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"
Sep 17 17:47:52.005040 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:47:52.005016    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460\": container with ID starting with e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460 not found: ID does not exist" containerID="e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"
Sep 17 17:47:52.005157 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.005050    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460"} err="failed to get container status \"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460\": rpc error: code = NotFound desc = could not find container \"e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460\": container with ID starting with e362aff16aaee165ddfb43f4b963a19b7dfa4a5a46eb3e3594f9fb1803270460 not found: ID does not exist"
Sep 17 17:47:52.006583 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.006565    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-njrqf"]
Sep 17 17:47:52.776780 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:52.776752    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" path="/var/lib/kubelet/pods/147f60e0-4a29-4a41-938c-f760898b17ce/volumes"
Sep 17 17:47:55.976587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:55.976555    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:47:55.976976 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:55.976796    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" containerID="cri-o://e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372" gracePeriod=30
Sep 17 17:47:55.976976 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:55.976841    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kube-rbac-proxy" containerID="cri-o://7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3" gracePeriod=30
Sep 17 17:47:57.002510 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:57.002480    2575 generic.go:356] "Generic (PLEG): container finished" podID="29667b89-38ad-4535-8b59-532289ebf1af" containerID="7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3" exitCode=2
Sep 17 17:47:57.002891 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:57.002515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerDied","Data":"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"}
Sep 17 17:47:57.690166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:57.690124    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 17:47:57.695472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:57.695448    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:47:59.510779 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.510756    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:47:59.617893 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.617831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls\") pod \"29667b89-38ad-4535-8b59-532289ebf1af\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") "
Sep 17 17:47:59.617893 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.617876    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config\") pod \"29667b89-38ad-4535-8b59-532289ebf1af\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") "
Sep 17 17:47:59.618025 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.617900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bqrs\" (UniqueName: \"kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs\") pod \"29667b89-38ad-4535-8b59-532289ebf1af\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") "
Sep 17 17:47:59.618025 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.617922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location\") pod \"29667b89-38ad-4535-8b59-532289ebf1af\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") "
Sep 17 17:47:59.618025 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.617938    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert\") pod \"29667b89-38ad-4535-8b59-532289ebf1af\" (UID: \"29667b89-38ad-4535-8b59-532289ebf1af\") "
Sep 17 17:47:59.618275 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.618244    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29667b89-38ad-4535-8b59-532289ebf1af" (UID: "29667b89-38ad-4535-8b59-532289ebf1af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:59.618370 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.618278    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29667b89-38ad-4535-8b59-532289ebf1af" (UID: "29667b89-38ad-4535-8b59-532289ebf1af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:59.618370 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.618285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29667b89-38ad-4535-8b59-532289ebf1af" (UID: "29667b89-38ad-4535-8b59-532289ebf1af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:59.619795 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.619771    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29667b89-38ad-4535-8b59-532289ebf1af" (UID: "29667b89-38ad-4535-8b59-532289ebf1af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:59.619890 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.619867    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs" (OuterVolumeSpecName: "kube-api-access-7bqrs") pod "29667b89-38ad-4535-8b59-532289ebf1af" (UID: "29667b89-38ad-4535-8b59-532289ebf1af"). InnerVolumeSpecName "kube-api-access-7bqrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:59.719053 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.719019    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29667b89-38ad-4535-8b59-532289ebf1af-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:59.719053 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.719050    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:59.719053 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.719060    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqrs\" (UniqueName: \"kubernetes.io/projected/29667b89-38ad-4535-8b59-532289ebf1af-kube-api-access-7bqrs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:59.719220 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.719070    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29667b89-38ad-4535-8b59-532289ebf1af-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:47:59.719220 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:47:59.719079    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29667b89-38ad-4535-8b59-532289ebf1af-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:48:00.014218 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.014167    2575 generic.go:356] "Generic (PLEG): container finished" podID="29667b89-38ad-4535-8b59-532289ebf1af" containerID="e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372" exitCode=0
Sep 17 17:48:00.014218 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.014209    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerDied","Data":"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"}
Sep 17 17:48:00.014341 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.014260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v" event={"ID":"29667b89-38ad-4535-8b59-532289ebf1af","Type":"ContainerDied","Data":"c57f7a127a41070634242b86f4dfeac6fd1dab5ee70a725bebfe0cfe9bf1987d"}
Sep 17 17:48:00.014341 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.014213    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"
Sep 17 17:48:00.014404 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.014299    2575 scope.go:117] "RemoveContainer" containerID="7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"
Sep 17 17:48:00.023934 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.023793    2575 scope.go:117] "RemoveContainer" containerID="e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"
Sep 17 17:48:00.030690 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.030674    2575 scope.go:117] "RemoveContainer" containerID="fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"
Sep 17 17:48:00.037184 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037169    2575 scope.go:117] "RemoveContainer" containerID="7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"
Sep 17 17:48:00.037440 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:48:00.037424    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3\": container with ID starting with 7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3 not found: ID does not exist" containerID="7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"
Sep 17 17:48:00.037498 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037447    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3"} err="failed to get container status \"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3\": rpc error: code = NotFound desc = could not find container \"7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3\": container with ID starting with 7293b91f1d6cb19d1f9651d24ce9aa948281d3b98dfea1e7bd68da0b96c315d3 not found: ID does not exist"
Sep 17 17:48:00.037498 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037465    2575 scope.go:117] "RemoveContainer" containerID="e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"
Sep 17 17:48:00.037690 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:48:00.037673    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372\": container with ID starting with e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372 not found: ID does not exist" containerID="e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"
Sep 17 17:48:00.037737 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037694    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372"} err="failed to get container status \"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372\": rpc error: code = NotFound desc = could not find container \"e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372\": container with ID starting with e0493a110eb6f96ed0723b78ac28fb3133d09c501b358a26a446316e38c5d372 not found: ID does not exist"
Sep 17 17:48:00.037737 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037710    2575 scope.go:117] "RemoveContainer" containerID="fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"
Sep 17 17:48:00.037956 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:48:00.037936    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b\": container with ID starting with fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b not found: ID does not exist" containerID="fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"
Sep 17 17:48:00.038008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.037973    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b"} err="failed to get container status \"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b\": rpc error: code = NotFound desc = could not find container \"fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b\": container with ID starting with fe0f5c2bd9e2e491c7a74c335cc9eebb5438874b9fe784a98ecb59117f0be73b not found: ID does not exist"
Sep 17 17:48:00.039456 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.039437    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:48:00.044289 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.044270    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-kfn7v"]
Sep 17 17:48:00.776451 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:00.776431    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29667b89-38ad-4535-8b59-532289ebf1af" path="/var/lib/kubelet/pods/29667b89-38ad-4535-8b59-532289ebf1af/volumes"
Sep 17 17:48:06.273071 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273035    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273295    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273305    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273314    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273319    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273328    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273334    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273343    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273349    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273357    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273362    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="storage-initializer"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273369    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273374    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273380    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273385    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273395    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273400    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273409    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273414    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273452    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kserve-container"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273459    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38329cad-31ab-49e1-87a2-0ed33be53588" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273466    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kube-rbac-proxy"
Sep 17 17:48:06.273461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273472    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kserve-container"
Sep 17 17:48:06.274126 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273476    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="147f60e0-4a29-4a41-938c-f760898b17ce" containerName="kserve-container"
Sep 17 17:48:06.274126 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.273482    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="29667b89-38ad-4535-8b59-532289ebf1af" containerName="kube-rbac-proxy"
Sep 17 17:48:06.276266 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.276248    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.278586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.278560    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:48:06.278691 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.278595    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-46f911-predictor-serving-cert\""
Sep 17 17:48:06.278691 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.278596    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:48:06.278810 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.278795    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:48:06.279538 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.279514    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-46f911-kube-rbac-proxy-sar-config\""
Sep 17 17:48:06.279634 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.279560    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:48:06.279634 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.279622    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:48:06.286303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.286282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:48:06.366652 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.366633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.366754 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.366660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.366754 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.366692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.366855 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.366767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.366855 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.366808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9gmr\" (UniqueName: \"kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.467895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.467872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9gmr\" (UniqueName: \"kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.467981 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.467900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.467981 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.467919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.467981 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.467956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.468138 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.467985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.468461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.468440    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.468574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.468558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.468660 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.468644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.470384 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.470362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.480497 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.480475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9gmr\" (UniqueName: \"kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr\") pod \"isvc-primary-46f911-predictor-567dd9546b-7b4jq\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") " pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.586705 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.586682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:06.864398 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:06.864331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:48:06.867005 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:48:06.866978    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74999a12_5ea7_4a13_b49c_3cca1721712b.slice/crio-21364b66ab6a1c321d3698c1183dc015d2f11176c52843cddccac6a263d700ed WatchSource:0}: Error finding container 21364b66ab6a1c321d3698c1183dc015d2f11176c52843cddccac6a263d700ed: Status 404 returned error can't find the container with id 21364b66ab6a1c321d3698c1183dc015d2f11176c52843cddccac6a263d700ed
Sep 17 17:48:07.038712 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:07.038677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerStarted","Data":"4e1f17f1f3064e569b70fb077724c4e65181b2cb99767122450a614a38b97622"}
Sep 17 17:48:07.038832 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:07.038731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerStarted","Data":"21364b66ab6a1c321d3698c1183dc015d2f11176c52843cddccac6a263d700ed"}
Sep 17 17:48:11.052574 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:11.052545    2575 generic.go:356] "Generic (PLEG): container finished" podID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerID="4e1f17f1f3064e569b70fb077724c4e65181b2cb99767122450a614a38b97622" exitCode=0
Sep 17 17:48:11.052854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:11.052626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerDied","Data":"4e1f17f1f3064e569b70fb077724c4e65181b2cb99767122450a614a38b97622"}
Sep 17 17:48:12.060229 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.059181    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerStarted","Data":"4f1c8a84283e33efb9c951d2b676717460446792184ba4c02287a058722b2e3c"}
Sep 17 17:48:12.060229 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.059289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerStarted","Data":"101da717e525e552a1281df7613720a2ea5bbfa6bb8d9e63a8b140b1ed5fd5a8"}
Sep 17 17:48:12.060229 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.060178    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:12.060229 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.060231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:12.061469 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.061441    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:12.082794 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:12.082757    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podStartSLOduration=2.106116523 podStartE2EDuration="6.082743364s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.976626841s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:12.081761639 +0000 UTC m=+863.448893542" watchObservedRunningTime="2026-09-17 17:48:12.082743364 +0000 UTC m=+863.449875245"
Sep 17 17:48:13.063618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:13.063574    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:18.067607 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:18.067579    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:48:18.068023 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:18.067999    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:28.068180 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:28.068144    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:36.572998 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.572914    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:48:36.576347 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.576325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.578665 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.578645    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4165a-predictor-serving-cert\""
Sep 17 17:48:36.578765 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.578696    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4165a-kube-rbac-proxy-sar-config\""
Sep 17 17:48:36.586372 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.586350    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:48:36.677346 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.677325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dz98\" (UniqueName: \"kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.677436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.677357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.677436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.677375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.778581 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.778560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dz98\" (UniqueName: \"kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.778678 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.778596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.778678 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.778616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.779263 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.779244    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.780806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.780785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.787473 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.787453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dz98\" (UniqueName: \"kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98\") pod \"message-dumper-raw-4165a-predictor-6d877867d9-2gk42\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") " pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:36.916386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:36.916361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:37.182876 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:37.182808    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:48:37.185771 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:48:37.185745    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a091bd_a05d_41f1_8a6c_155d9f0a9b83.slice/crio-8da03ce3115c42a522bea3c2e5f9a44c075e106ef4e77d47ed2a2ca1cbdf16b0 WatchSource:0}: Error finding container 8da03ce3115c42a522bea3c2e5f9a44c075e106ef4e77d47ed2a2ca1cbdf16b0: Status 404 returned error can't find the container with id 8da03ce3115c42a522bea3c2e5f9a44c075e106ef4e77d47ed2a2ca1cbdf16b0
Sep 17 17:48:37.187614 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:37.187599    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:48:38.069250 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:38.069150    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:38.159178 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:38.159137    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerStarted","Data":"8da03ce3115c42a522bea3c2e5f9a44c075e106ef4e77d47ed2a2ca1cbdf16b0"}
Sep 17 17:48:39.164426 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:39.164387    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerStarted","Data":"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"}
Sep 17 17:48:39.164790 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:39.164456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerStarted","Data":"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"}
Sep 17 17:48:39.164790 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:39.164555    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:39.186989 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:39.186945    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" podStartSLOduration=2.065193576 podStartE2EDuration="3.186933273s" totalImagesPullingTime="1.121739697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:39.185699684 +0000 UTC m=+890.552831562" watchObservedRunningTime="2026-09-17 17:48:39.186933273 +0000 UTC m=+890.554065153"
Sep 17 17:48:40.168570 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:40.168538    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:40.169979 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:40.169955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:47.180880 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:47.180851    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:48:48.068829 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:48.068790    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:50.736867 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:50.736834    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:48:50.738003 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:50.737984    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:48:56.569745 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.569714    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:48:56.577534 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.577509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.579781 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.579761    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4165a-predictor-serving-cert\""
Sep 17 17:48:56.579886 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.579803    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4165a-kube-rbac-proxy-sar-config\""
Sep 17 17:48:56.587653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.587629    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:48:56.716179 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.716156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.716303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.716185    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.716303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.716221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.716303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.716239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvxq\" (UniqueName: \"kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.716303 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.716275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.816724 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.816702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.816845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.816730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.816845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.816747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.816845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.816781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.816845 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.816801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvxq\" (UniqueName: \"kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.817179 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.817158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.817453 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.817432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.817515 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.817450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.819260 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.819243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.825833 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.825788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvxq\" (UniqueName: \"kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq\") pod \"isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:56.889299 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:56.889276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:48:57.162063 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:57.162030    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:48:57.165556 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:48:57.165533    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69c1a50_9e04_4594_85b0_6b66e8e1a538.slice/crio-bfe454786e5e05cb34e8bdfe9a3344724f83294b9e3c0314b807030caa89a1d4 WatchSource:0}: Error finding container bfe454786e5e05cb34e8bdfe9a3344724f83294b9e3c0314b807030caa89a1d4: Status 404 returned error can't find the container with id bfe454786e5e05cb34e8bdfe9a3344724f83294b9e3c0314b807030caa89a1d4
Sep 17 17:48:57.230727 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:57.230698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerStarted","Data":"bfe454786e5e05cb34e8bdfe9a3344724f83294b9e3c0314b807030caa89a1d4"}
Sep 17 17:48:58.068353 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:58.068277    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:48:58.236215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:48:58.236179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerStarted","Data":"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"}
Sep 17 17:49:01.248689 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:01.248660    2575 generic.go:356] "Generic (PLEG): container finished" podID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerID="6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55" exitCode=0
Sep 17 17:49:01.249007 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:01.248735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerDied","Data":"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"}
Sep 17 17:49:02.254728 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:02.254696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerStarted","Data":"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"}
Sep 17 17:49:02.255082 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:02.254759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerStarted","Data":"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"}
Sep 17 17:49:02.255082 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:02.254786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerStarted","Data":"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"}
Sep 17 17:49:02.255082 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:02.254984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:49:02.276525 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:02.276483    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podStartSLOduration=2.296584693 podStartE2EDuration="6.276468576s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.979883883s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:02.275460303 +0000 UTC m=+913.642592186" watchObservedRunningTime="2026-09-17 17:49:02.276468576 +0000 UTC m=+913.643600450"
Sep 17 17:49:03.258482 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:03.258445    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:49:03.258934 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:03.258520    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:49:03.259910 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:03.259871    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:03.260502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:03.260477    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:04.262260 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:04.262218    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:04.262673 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:04.262644    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:08.068781 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:08.068745    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:49:09.266649 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:09.266619    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:49:09.267261 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:09.267235    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:09.267543 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:09.267514    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:18.068843 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:18.068805    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:49:19.267864 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:19.267825    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:19.268391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:19.268363    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:28.069154 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:28.069114    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:49:29.267897 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:29.267850    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:29.268386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:29.268320    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:39.267725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:39.267685    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:39.268313 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:39.268177    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:49.267718 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:49.267672    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:49.268226 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:49.268200    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:59.267222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:59.267176    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:49:59.267725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:49:59.267661    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:05.498614 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:05.498529    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:50:05.499042 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:05.498822    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" containerID="cri-o://101da717e525e552a1281df7613720a2ea5bbfa6bb8d9e63a8b140b1ed5fd5a8" gracePeriod=30
Sep 17 17:50:05.499042 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:05.498894    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kube-rbac-proxy" containerID="cri-o://4f1c8a84283e33efb9c951d2b676717460446792184ba4c02287a058722b2e3c" gracePeriod=30
Sep 17 17:50:06.495783 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:06.495751    2575 generic.go:356] "Generic (PLEG): container finished" podID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerID="4f1c8a84283e33efb9c951d2b676717460446792184ba4c02287a058722b2e3c" exitCode=2
Sep 17 17:50:06.495949 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:06.495824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerDied","Data":"4f1c8a84283e33efb9c951d2b676717460446792184ba4c02287a058722b2e3c"}
Sep 17 17:50:08.064141 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:08.064077    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 17 17:50:08.068222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:08.068190    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 17:50:09.267215 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.267171    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:50:09.267623 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.267544    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:09.513606 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.513571    2575 generic.go:356] "Generic (PLEG): container finished" podID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerID="101da717e525e552a1281df7613720a2ea5bbfa6bb8d9e63a8b140b1ed5fd5a8" exitCode=0
Sep 17 17:50:09.513784 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.513758    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerDied","Data":"101da717e525e552a1281df7613720a2ea5bbfa6bb8d9e63a8b140b1ed5fd5a8"}
Sep 17 17:50:09.553323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.553299    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:50:09.638461 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638434    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9gmr\" (UniqueName: \"kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr\") pod \"74999a12-5ea7-4a13-b49c-3cca1721712b\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") "
Sep 17 17:50:09.638586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638483    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert\") pod \"74999a12-5ea7-4a13-b49c-3cca1721712b\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") "
Sep 17 17:50:09.638586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638522    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location\") pod \"74999a12-5ea7-4a13-b49c-3cca1721712b\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") "
Sep 17 17:50:09.638586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638548    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config\") pod \"74999a12-5ea7-4a13-b49c-3cca1721712b\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") "
Sep 17 17:50:09.638586 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638573    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls\") pod \"74999a12-5ea7-4a13-b49c-3cca1721712b\" (UID: \"74999a12-5ea7-4a13-b49c-3cca1721712b\") "
Sep 17 17:50:09.638877 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638853    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74999a12-5ea7-4a13-b49c-3cca1721712b" (UID: "74999a12-5ea7-4a13-b49c-3cca1721712b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:09.638877 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638862    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74999a12-5ea7-4a13-b49c-3cca1721712b" (UID: "74999a12-5ea7-4a13-b49c-3cca1721712b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:09.638975 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.638893    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74999a12-5ea7-4a13-b49c-3cca1721712b" (UID: "74999a12-5ea7-4a13-b49c-3cca1721712b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:09.640470 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.640446    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74999a12-5ea7-4a13-b49c-3cca1721712b" (UID: "74999a12-5ea7-4a13-b49c-3cca1721712b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:09.640553 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.640536    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr" (OuterVolumeSpecName: "kube-api-access-f9gmr") pod "74999a12-5ea7-4a13-b49c-3cca1721712b" (UID: "74999a12-5ea7-4a13-b49c-3cca1721712b"). InnerVolumeSpecName "kube-api-access-f9gmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:09.739279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.739257    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:09.739279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.739278    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74999a12-5ea7-4a13-b49c-3cca1721712b-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:09.739399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.739289    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9gmr\" (UniqueName: \"kubernetes.io/projected/74999a12-5ea7-4a13-b49c-3cca1721712b-kube-api-access-f9gmr\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:09.739399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.739297    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74999a12-5ea7-4a13-b49c-3cca1721712b-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:09.739399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:09.739305    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74999a12-5ea7-4a13-b49c-3cca1721712b-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:10.520250 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.520214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq" event={"ID":"74999a12-5ea7-4a13-b49c-3cca1721712b","Type":"ContainerDied","Data":"21364b66ab6a1c321d3698c1183dc015d2f11176c52843cddccac6a263d700ed"}
Sep 17 17:50:10.520718 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.520266    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"
Sep 17 17:50:10.520718 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.520295    2575 scope.go:117] "RemoveContainer" containerID="4f1c8a84283e33efb9c951d2b676717460446792184ba4c02287a058722b2e3c"
Sep 17 17:50:10.529550 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.529526    2575 scope.go:117] "RemoveContainer" containerID="101da717e525e552a1281df7613720a2ea5bbfa6bb8d9e63a8b140b1ed5fd5a8"
Sep 17 17:50:10.537120 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.537087    2575 scope.go:117] "RemoveContainer" containerID="4e1f17f1f3064e569b70fb077724c4e65181b2cb99767122450a614a38b97622"
Sep 17 17:50:10.552270 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.552248    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:50:10.557251 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.557229    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-46f911-predictor-567dd9546b-7b4jq"]
Sep 17 17:50:10.778982 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:10.778904    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" path="/var/lib/kubelet/pods/74999a12-5ea7-4a13-b49c-3cca1721712b/volumes"
Sep 17 17:50:19.268291 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:19.268258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:50:19.268755 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:19.268383    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:50:31.550703 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.550668    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4165a-predictor-6d877867d9-2gk42_44a091bd-a05d-41f1-8a6c-155d9f0a9b83/kserve-container/0.log"
Sep 17 17:50:31.766190 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.766155    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:50:31.766523 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.766477    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" containerID="cri-o://4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6" gracePeriod=30
Sep 17 17:50:31.766632 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.766520    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" containerID="cri-o://a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21" gracePeriod=30
Sep 17 17:50:31.766698 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.766512    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" containerID="cri-o://09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58" gracePeriod=30
Sep 17 17:50:31.790410 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.790385    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:50:31.790566 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.790550    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kserve-container" containerID="cri-o://8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e" gracePeriod=30
Sep 17 17:50:31.790682 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.790644    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kube-rbac-proxy" containerID="cri-o://416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a" gracePeriod=30
Sep 17 17:50:31.926369 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926344    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:50:31.926619 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926608    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kube-rbac-proxy"
Sep 17 17:50:31.926662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926620    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kube-rbac-proxy"
Sep 17 17:50:31.926662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926633    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container"
Sep 17 17:50:31.926662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926639    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container"
Sep 17 17:50:31.926662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926651    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="storage-initializer"
Sep 17 17:50:31.926662 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926658    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="storage-initializer"
Sep 17 17:50:31.926806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926704    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kube-rbac-proxy"
Sep 17 17:50:31.926806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.926711    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74999a12-5ea7-4a13-b49c-3cca1721712b" containerName="kserve-container"
Sep 17 17:50:31.929629 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.929612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:31.932398 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.932382    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-75f8f-predictor-serving-cert\""
Sep 17 17:50:31.932479 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.932442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-75f8f-kube-rbac-proxy-sar-config\""
Sep 17 17:50:31.943375 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:31.943351    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:50:32.039391 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.039371    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:50:32.092018 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.091993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.092141 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.092029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.092141 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.092062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.092141 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.092092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5j7p\" (UniqueName: \"kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.092141 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.092128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.192909 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.192888    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config\") pod \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") "
Sep 17 17:50:32.193012 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.192927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dz98\" (UniqueName: \"kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98\") pod \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") "
Sep 17 17:50:32.193012 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.192956    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls\") pod \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\" (UID: \"44a091bd-a05d-41f1-8a6c-155d9f0a9b83\") "
Sep 17 17:50:32.193081 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193081 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193175 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193175 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5j7p\" (UniqueName: \"kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193175 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193313 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193283    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44a091bd-a05d-41f1-8a6c-155d9f0a9b83" (UID: "44a091bd-a05d-41f1-8a6c-155d9f0a9b83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:32.193520 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.193870 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.193839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.195612 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.195583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44a091bd-a05d-41f1-8a6c-155d9f0a9b83" (UID: "44a091bd-a05d-41f1-8a6c-155d9f0a9b83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:32.195697 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.195605    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98" (OuterVolumeSpecName: "kube-api-access-8dz98") pod "44a091bd-a05d-41f1-8a6c-155d9f0a9b83" (UID: "44a091bd-a05d-41f1-8a6c-155d9f0a9b83"). InnerVolumeSpecName "kube-api-access-8dz98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:32.195697 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.195642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.210193 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.210174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5j7p\" (UniqueName: \"kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p\") pod \"isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.240152 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.240131    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:32.294540 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.294466    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:32.294540 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.294503    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dz98\" (UniqueName: \"kubernetes.io/projected/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-kube-api-access-8dz98\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:32.294540 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.294517    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a091bd-a05d-41f1-8a6c-155d9f0a9b83-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:50:32.513712 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.513685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:50:32.516058 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:50:32.516028    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44a0410_ec3b_4e3e_99b1_f91b34ca3012.slice/crio-86c7e153cd915b6efa50229dd9347d7f76fd114dc4318c189e881d487e032ee9 WatchSource:0}: Error finding container 86c7e153cd915b6efa50229dd9347d7f76fd114dc4318c189e881d487e032ee9: Status 404 returned error can't find the container with id 86c7e153cd915b6efa50229dd9347d7f76fd114dc4318c189e881d487e032ee9
Sep 17 17:50:32.609918 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.609892    2575 generic.go:356] "Generic (PLEG): container finished" podID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerID="a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21" exitCode=2
Sep 17 17:50:32.610249 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.609970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerDied","Data":"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"}
Sep 17 17:50:32.611653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611624    2575 generic.go:356] "Generic (PLEG): container finished" podID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerID="416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a" exitCode=2
Sep 17 17:50:32.611653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611649    2575 generic.go:356] "Generic (PLEG): container finished" podID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerID="8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e" exitCode=2
Sep 17 17:50:32.611818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611671    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"
Sep 17 17:50:32.611818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerDied","Data":"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"}
Sep 17 17:50:32.611818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerDied","Data":"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"}
Sep 17 17:50:32.611818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42" event={"ID":"44a091bd-a05d-41f1-8a6c-155d9f0a9b83","Type":"ContainerDied","Data":"8da03ce3115c42a522bea3c2e5f9a44c075e106ef4e77d47ed2a2ca1cbdf16b0"}
Sep 17 17:50:32.612014 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.611845    2575 scope.go:117] "RemoveContainer" containerID="416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"
Sep 17 17:50:32.613402 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.613377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerStarted","Data":"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"}
Sep 17 17:50:32.613484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.613447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerStarted","Data":"86c7e153cd915b6efa50229dd9347d7f76fd114dc4318c189e881d487e032ee9"}
Sep 17 17:50:32.621647 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.621629    2575 scope.go:117] "RemoveContainer" containerID="8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"
Sep 17 17:50:32.629545 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.629526    2575 scope.go:117] "RemoveContainer" containerID="416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"
Sep 17 17:50:32.629815 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:50:32.629796    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a\": container with ID starting with 416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a not found: ID does not exist" containerID="416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"
Sep 17 17:50:32.629902 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.629825    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"} err="failed to get container status \"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a\": rpc error: code = NotFound desc = could not find container \"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a\": container with ID starting with 416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a not found: ID does not exist"
Sep 17 17:50:32.629902 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.629855    2575 scope.go:117] "RemoveContainer" containerID="8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"
Sep 17 17:50:32.630168 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:50:32.630146    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e\": container with ID starting with 8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e not found: ID does not exist" containerID="8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"
Sep 17 17:50:32.630484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.630173    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"} err="failed to get container status \"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e\": rpc error: code = NotFound desc = could not find container \"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e\": container with ID starting with 8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e not found: ID does not exist"
Sep 17 17:50:32.630484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.630196    2575 scope.go:117] "RemoveContainer" containerID="416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"
Sep 17 17:50:32.630484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.630461    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a"} err="failed to get container status \"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a\": rpc error: code = NotFound desc = could not find container \"416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a\": container with ID starting with 416ee72737c207217cbe66ca254c6653662c201408c8f7e7671603149bd9ad8a not found: ID does not exist"
Sep 17 17:50:32.630484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.630478    2575 scope.go:117] "RemoveContainer" containerID="8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"
Sep 17 17:50:32.630757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.630732    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e"} err="failed to get container status \"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e\": rpc error: code = NotFound desc = could not find container \"8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e\": container with ID starting with 8449bbf299f3aefd0d7d2208f67c47fa99fcc93308deb295e92df5e18aef818e not found: ID does not exist"
Sep 17 17:50:32.658086 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.658063    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:50:32.659501 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.659479    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4165a-predictor-6d877867d9-2gk42"]
Sep 17 17:50:32.778441 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:32.778371    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" path="/var/lib/kubelet/pods/44a091bd-a05d-41f1-8a6c-155d9f0a9b83/volumes"
Sep 17 17:50:34.262658 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:34.262618    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:35.628679 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:35.628652    2575 generic.go:356] "Generic (PLEG): container finished" podID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerID="4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6" exitCode=0
Sep 17 17:50:35.628966 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:35.628726    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerDied","Data":"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"}
Sep 17 17:50:36.633850 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:36.633811    2575 generic.go:356] "Generic (PLEG): container finished" podID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerID="bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49" exitCode=0
Sep 17 17:50:36.634197 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:36.633888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerDied","Data":"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"}
Sep 17 17:50:37.639957 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:37.639923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerStarted","Data":"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"}
Sep 17 17:50:37.640334 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:37.639981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerStarted","Data":"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"}
Sep 17 17:50:37.640334 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:37.640165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:37.661801 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:37.661753    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podStartSLOduration=2.9066067479999997 podStartE2EDuration="6.6617353s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755128552s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:37.66035954 +0000 UTC m=+1009.027491421" watchObservedRunningTime="2026-09-17 17:50:37.6617353 +0000 UTC m=+1009.028867184"
Sep 17 17:50:38.644008 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:38.643980    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:38.644984 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:38.644960    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:50:39.263187 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:39.263156    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:39.267538 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:39.267514    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:50:39.267853 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:39.267832    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:39.649741 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:39.649710    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:50:44.262475 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:44.262432    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:44.262964 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:44.262547    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:50:44.654771 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:44.654750    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:50:44.655234 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:44.655208    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:50:49.262966 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:49.262929    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:49.267318 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:49.267284    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:50:49.267576 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:49.267558    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:54.262336 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:54.262298    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:54.655205 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:54.655169    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:50:59.263312 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:59.263272    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:50:59.267618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:59.267592    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:50:59.267722 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:59.267707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:50:59.267978 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:59.267955    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:59.268136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:50:59.268039    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:51:01.907063 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:01.907036    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:51:02.090239 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090211    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls\") pod \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") "
Sep 17 17:51:02.090373 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090268    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config\") pod \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") "
Sep 17 17:51:02.090373 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090290    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert\") pod \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") "
Sep 17 17:51:02.090373 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090340    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location\") pod \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") "
Sep 17 17:51:02.090536 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090386    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lvxq\" (UniqueName: \"kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq\") pod \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\" (UID: \"b69c1a50-9e04-4594-85b0-6b66e8e1a538\") "
Sep 17 17:51:02.090658 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090634    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b69c1a50-9e04-4594-85b0-6b66e8e1a538" (UID: "b69c1a50-9e04-4594-85b0-6b66e8e1a538"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:02.090725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090646    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b69c1a50-9e04-4594-85b0-6b66e8e1a538" (UID: "b69c1a50-9e04-4594-85b0-6b66e8e1a538"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:02.090725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.090680    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b69c1a50-9e04-4594-85b0-6b66e8e1a538" (UID: "b69c1a50-9e04-4594-85b0-6b66e8e1a538"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:02.092284 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.092254    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b69c1a50-9e04-4594-85b0-6b66e8e1a538" (UID: "b69c1a50-9e04-4594-85b0-6b66e8e1a538"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:02.092394 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.092296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq" (OuterVolumeSpecName: "kube-api-access-9lvxq") pod "b69c1a50-9e04-4594-85b0-6b66e8e1a538" (UID: "b69c1a50-9e04-4594-85b0-6b66e8e1a538"). InnerVolumeSpecName "kube-api-access-9lvxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:02.191661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.191632    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b69c1a50-9e04-4594-85b0-6b66e8e1a538-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:51:02.191661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.191660    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:51:02.191661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.191669    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b69c1a50-9e04-4594-85b0-6b66e8e1a538-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:51:02.191825 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.191679    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:51:02.191825 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.191686    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvxq\" (UniqueName: \"kubernetes.io/projected/b69c1a50-9e04-4594-85b0-6b66e8e1a538-kube-api-access-9lvxq\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:51:02.729268 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.729235    2575 generic.go:356] "Generic (PLEG): container finished" podID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerID="09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58" exitCode=0
Sep 17 17:51:02.729424 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.729295    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"
Sep 17 17:51:02.729424 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.729309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerDied","Data":"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"}
Sep 17 17:51:02.729424 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.729384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl" event={"ID":"b69c1a50-9e04-4594-85b0-6b66e8e1a538","Type":"ContainerDied","Data":"bfe454786e5e05cb34e8bdfe9a3344724f83294b9e3c0314b807030caa89a1d4"}
Sep 17 17:51:02.729530 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.729427    2575 scope.go:117] "RemoveContainer" containerID="09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"
Sep 17 17:51:02.737541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.737522    2575 scope.go:117] "RemoveContainer" containerID="a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"
Sep 17 17:51:02.745250 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.745172    2575 scope.go:117] "RemoveContainer" containerID="4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"
Sep 17 17:51:02.754683 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.754656    2575 scope.go:117] "RemoveContainer" containerID="6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"
Sep 17 17:51:02.755218 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.755196    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:51:02.760853 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.760833    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4165a-predictor-c96cc7bb7-lnwxl"]
Sep 17 17:51:02.762066 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762051    2575 scope.go:117] "RemoveContainer" containerID="09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"
Sep 17 17:51:02.762340 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:51:02.762325    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58\": container with ID starting with 09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58 not found: ID does not exist" containerID="09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"
Sep 17 17:51:02.762401 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762348    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58"} err="failed to get container status \"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58\": rpc error: code = NotFound desc = could not find container \"09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58\": container with ID starting with 09ec6c684197777df941efb16f27d4691b5e7248db2f79565a37feecc7a4ea58 not found: ID does not exist"
Sep 17 17:51:02.762401 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762364    2575 scope.go:117] "RemoveContainer" containerID="a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"
Sep 17 17:51:02.762590 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:51:02.762571    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21\": container with ID starting with a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21 not found: ID does not exist" containerID="a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"
Sep 17 17:51:02.762636 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762596    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21"} err="failed to get container status \"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21\": rpc error: code = NotFound desc = could not find container \"a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21\": container with ID starting with a22f539b51844b62e586c5c4696dbcdaee80fb282aab4f73a243479fe271db21 not found: ID does not exist"
Sep 17 17:51:02.762636 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762611    2575 scope.go:117] "RemoveContainer" containerID="4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"
Sep 17 17:51:02.762867 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:51:02.762846    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6\": container with ID starting with 4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6 not found: ID does not exist" containerID="4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"
Sep 17 17:51:02.762932 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762878    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6"} err="failed to get container status \"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6\": rpc error: code = NotFound desc = could not find container \"4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6\": container with ID starting with 4fe6db344c67a422a1a59b6b34b7e296701c6c4298de8e6ccaf93b530be61de6 not found: ID does not exist"
Sep 17 17:51:02.762932 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.762901    2575 scope.go:117] "RemoveContainer" containerID="6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"
Sep 17 17:51:02.763158 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:51:02.763141    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55\": container with ID starting with 6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55 not found: ID does not exist" containerID="6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"
Sep 17 17:51:02.763200 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.763162    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55"} err="failed to get container status \"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55\": rpc error: code = NotFound desc = could not find container \"6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55\": container with ID starting with 6d7e5573c34d4d73566fb5ae6b8b3f845200491d5ebc673320bb72fb825c5b55 not found: ID does not exist"
Sep 17 17:51:02.776837 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:02.776818    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" path="/var/lib/kubelet/pods/b69c1a50-9e04-4594-85b0-6b66e8e1a538/volumes"
Sep 17 17:51:04.656054 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:04.656021    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:51:14.655801 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:14.655759    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:51:24.655253 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:24.655218    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:51:34.655381 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:34.655294    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:51:44.656210 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:44.656171    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:51:54.655916 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:51:54.655881    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:04.655807 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:04.655769    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:08.772753 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:08.772704    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:18.773110 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:18.773052    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:28.772955 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:28.772905    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:38.773541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:38.773494    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:48.773176 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:48.773123    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:52:58.772759 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:52:58.772713    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:53:08.776813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:08.776726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:53:11.995124 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:11.995078    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:53:11.995569 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:11.995414    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" containerID="cri-o://139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052" gracePeriod=30
Sep 17 17:53:11.995569 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:11.995440    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy" containerID="cri-o://58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7" gracePeriod=30
Sep 17 17:53:12.173205 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:12.173181    2575 generic.go:356] "Generic (PLEG): container finished" podID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerID="58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7" exitCode=2
Sep 17 17:53:12.173306 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:12.173251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerDied","Data":"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"}
Sep 17 17:53:14.650640 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:14.650593    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 17:53:16.615812 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.615779    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616196    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616214    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616229    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616236    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616257    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="storage-initializer"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616267    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="storage-initializer"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616277    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent"
Sep 17 17:53:16.616279 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616285    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616297    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kserve-container"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616308    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kserve-container"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616326    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616334    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616435    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616447    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="agent"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616458    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kserve-container"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616508    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a091bd-a05d-41f1-8a6c-155d9f0a9b83" containerName="kube-rbac-proxy"
Sep 17 17:53:16.616709 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.616564    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69c1a50-9e04-4594-85b0-6b66e8e1a538" containerName="kserve-container"
Sep 17 17:53:16.619661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.619641    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.622670 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.622650    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:53:16.622769 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.622706    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 17:53:16.632139 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.632118    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:53:16.713252 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.713227    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:53:16.716320 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.716385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.716385 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.716464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.716464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsnx\" (UniqueName: \"kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.716464 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.716445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.718643 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.718624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:53:16.726796 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.726776    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:53:16.817525 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.817653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.817653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.817653 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.817836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsnx\" (UniqueName: \"kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.817836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.817836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.818005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.818005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2jp\" (UniqueName: \"kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.818005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.818005 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.817977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.818315 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.818297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.818645 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.818627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.820178 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.820162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.829482 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.829454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsnx\" (UniqueName: \"kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:16.918501 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.918501 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2jp\" (UniqueName: \"kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.918654 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.918654 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.918654 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.918654 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:53:16.918643    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:53:16.918862 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:53:16.918711    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls podName:6041bf58-8ed7-4c98-a125-75578c31effd nodeName:}" failed. No retries permitted until 2026-09-17 17:53:17.418692827 +0000 UTC m=+1168.785824688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls") pod "isvc-canary-create-predictor-7d998654b-l5p9b" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:53:16.918979 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.918960    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.919109 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.919076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.919267 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.919247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.927834 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.927809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2jp\" (UniqueName: \"kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:16.929476 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:16.929450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:17.201486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:17.201457    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:53:17.203795 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:53:17.203768    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2854300_a64d_430c_9d4c_0f1d5f32331d.slice/crio-c97912df32d66048a16088434c726b3e7aa4781b8cee0be59b0f2f5a0b430c36 WatchSource:0}: Error finding container c97912df32d66048a16088434c726b3e7aa4781b8cee0be59b0f2f5a0b430c36: Status 404 returned error can't find the container with id c97912df32d66048a16088434c726b3e7aa4781b8cee0be59b0f2f5a0b430c36
Sep 17 17:53:17.422412 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:17.422384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:17.424546 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:17.424521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-l5p9b\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:17.626914 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:17.626883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:17.897943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:17.897914    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:53:17.899289 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:53:17.899257    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6041bf58_8ed7_4c98_a125_75578c31effd.slice/crio-35903e7a6584f634b7f71299584e5aaa94ab9c2bdf40a59068393d405d900b4d WatchSource:0}: Error finding container 35903e7a6584f634b7f71299584e5aaa94ab9c2bdf40a59068393d405d900b4d: Status 404 returned error can't find the container with id 35903e7a6584f634b7f71299584e5aaa94ab9c2bdf40a59068393d405d900b4d
Sep 17 17:53:18.198140 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:18.198046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerStarted","Data":"31fd3e47e6bb65882bfef4592c0015632a7c83110d18c658f5a3471fb4aa3527"}
Sep 17 17:53:18.198291 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:18.198141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerStarted","Data":"c97912df32d66048a16088434c726b3e7aa4781b8cee0be59b0f2f5a0b430c36"}
Sep 17 17:53:18.199617 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:18.199592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerStarted","Data":"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"}
Sep 17 17:53:18.199754 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:18.199649    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerStarted","Data":"35903e7a6584f634b7f71299584e5aaa94ab9c2bdf40a59068393d405d900b4d"}
Sep 17 17:53:18.772704 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:18.772660    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 17:53:19.650198 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:19.650158    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 17:53:21.138031 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.138007    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:53:21.214408 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.214378    2575 generic.go:356] "Generic (PLEG): container finished" podID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerID="139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052" exitCode=0
Sep 17 17:53:21.214512 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.214432    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"
Sep 17 17:53:21.214512 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.214461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerDied","Data":"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"}
Sep 17 17:53:21.214602 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.214523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc" event={"ID":"d44a0410-ec3b-4e3e-99b1-f91b34ca3012","Type":"ContainerDied","Data":"86c7e153cd915b6efa50229dd9347d7f76fd114dc4318c189e881d487e032ee9"}
Sep 17 17:53:21.214602 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.214561    2575 scope.go:117] "RemoveContainer" containerID="58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"
Sep 17 17:53:21.216004 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.215983    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerID="31fd3e47e6bb65882bfef4592c0015632a7c83110d18c658f5a3471fb4aa3527" exitCode=0
Sep 17 17:53:21.216177 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.216036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerDied","Data":"31fd3e47e6bb65882bfef4592c0015632a7c83110d18c658f5a3471fb4aa3527"}
Sep 17 17:53:21.231776 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.231756    2575 scope.go:117] "RemoveContainer" containerID="139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"
Sep 17 17:53:21.242626 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.242610    2575 scope.go:117] "RemoveContainer" containerID="bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"
Sep 17 17:53:21.250940 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.250918    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config\") pod \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") "
Sep 17 17:53:21.251037 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.250985    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls\") pod \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") "
Sep 17 17:53:21.251037 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert\") pod \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") "
Sep 17 17:53:21.251156 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location\") pod \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") "
Sep 17 17:53:21.251224 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5j7p\" (UniqueName: \"kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p\") pod \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\" (UID: \"d44a0410-ec3b-4e3e-99b1-f91b34ca3012\") "
Sep 17 17:53:21.251376 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251346    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d44a0410-ec3b-4e3e-99b1-f91b34ca3012" (UID: "d44a0410-ec3b-4e3e-99b1-f91b34ca3012"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:21.251535 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d44a0410-ec3b-4e3e-99b1-f91b34ca3012" (UID: "d44a0410-ec3b-4e3e-99b1-f91b34ca3012"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:21.251612 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.251545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d44a0410-ec3b-4e3e-99b1-f91b34ca3012" (UID: "d44a0410-ec3b-4e3e-99b1-f91b34ca3012"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:21.253204 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.253179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d44a0410-ec3b-4e3e-99b1-f91b34ca3012" (UID: "d44a0410-ec3b-4e3e-99b1-f91b34ca3012"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:21.253283 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.253208    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p" (OuterVolumeSpecName: "kube-api-access-n5j7p") pod "d44a0410-ec3b-4e3e-99b1-f91b34ca3012" (UID: "d44a0410-ec3b-4e3e-99b1-f91b34ca3012"). InnerVolumeSpecName "kube-api-access-n5j7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:21.254739 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.254721    2575 scope.go:117] "RemoveContainer" containerID="58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"
Sep 17 17:53:21.255030 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:53:21.255003    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7\": container with ID starting with 58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7 not found: ID does not exist" containerID="58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"
Sep 17 17:53:21.255120 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.255039    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7"} err="failed to get container status \"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7\": rpc error: code = NotFound desc = could not find container \"58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7\": container with ID starting with 58d7e620fcdb9e72871091f204bb07f35ac7654c66d84399b2ecace73624d7a7 not found: ID does not exist"
Sep 17 17:53:21.255120 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.255055    2575 scope.go:117] "RemoveContainer" containerID="139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"
Sep 17 17:53:21.255457 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:53:21.255426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052\": container with ID starting with 139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052 not found: ID does not exist" containerID="139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"
Sep 17 17:53:21.255537 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.255461    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052"} err="failed to get container status \"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052\": rpc error: code = NotFound desc = could not find container \"139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052\": container with ID starting with 139b32347bfc2cc94f4ebb0a0998d1fe52460785ef5ed79db9498713b1afe052 not found: ID does not exist"
Sep 17 17:53:21.255537 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.255476    2575 scope.go:117] "RemoveContainer" containerID="bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"
Sep 17 17:53:21.255762 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:53:21.255741    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49\": container with ID starting with bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49 not found: ID does not exist" containerID="bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"
Sep 17 17:53:21.255834 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.255766    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49"} err="failed to get container status \"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49\": rpc error: code = NotFound desc = could not find container \"bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49\": container with ID starting with bddb376526eaf396fd15a86195a617984afe6ddc7dcf2438975ee04a15daea49 not found: ID does not exist"
Sep 17 17:53:21.351937 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.351912    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5j7p\" (UniqueName: \"kubernetes.io/projected/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-api-access-n5j7p\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:53:21.351937 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.351937    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:53:21.352128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.351948    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:53:21.352128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.351958    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:53:21.352128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.351966    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44a0410-ec3b-4e3e-99b1-f91b34ca3012-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:53:21.539362 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.539331    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:53:21.542318 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:21.542291    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-75f8f-predictor-f96b5bd9c-s6qdc"]
Sep 17 17:53:22.222335 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.222303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerStarted","Data":"6990877f6d646c371139d8cb2c65fec876fe7723bfec82586345cd97232c953b"}
Sep 17 17:53:22.222722 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.222378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerStarted","Data":"a7be253bf090678cdea5a8ac5f707343bd15ff603c798858ef58bbaf9f4f99d8"}
Sep 17 17:53:22.222722 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.222670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:22.224728 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.224703    2575 generic.go:356] "Generic (PLEG): container finished" podID="6041bf58-8ed7-4c98-a125-75578c31effd" containerID="ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968" exitCode=0
Sep 17 17:53:22.224860 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.224736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerDied","Data":"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"}
Sep 17 17:53:22.241733 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.241686    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podStartSLOduration=2.32731473 podStartE2EDuration="6.241666109s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914351379s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:22.240605644 +0000 UTC m=+1173.607737538" watchObservedRunningTime="2026-09-17 17:53:22.241666109 +0000 UTC m=+1173.608798048"
Sep 17 17:53:22.777725 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:22.777694    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" path="/var/lib/kubelet/pods/d44a0410-ec3b-4e3e-99b1-f91b34ca3012/volumes"
Sep 17 17:53:23.231804 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.231766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerStarted","Data":"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"}
Sep 17 17:53:23.232278 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.231834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerStarted","Data":"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"}
Sep 17 17:53:23.232278 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.232114    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:23.232436 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.232411    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:23.232505 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.232483    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:23.233174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.233152    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:23.233629 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.233609    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:53:23.252481 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:23.252430    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podStartSLOduration=3.403328131 podStartE2EDuration="7.252419557s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.849091426s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:23.250995362 +0000 UTC m=+1174.618127243" watchObservedRunningTime="2026-09-17 17:53:23.252419557 +0000 UTC m=+1174.619551439"
Sep 17 17:53:24.236937 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:24.236894    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:24.237421 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:24.236999    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:53:29.240991 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:29.240957    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:53:29.241437 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:29.241347    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:53:29.241437 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:29.241376    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:29.241749 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:29.241727    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:53:39.242204 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:39.242160    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:53:39.242204 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:39.242191    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:49.241974 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:49.241929    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:53:49.242439 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:49.241929    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:50.763911 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:50.763881    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:53:50.766058 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:50.766033    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:53:59.241488 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:59.241441    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:53:59.241927 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:53:59.241897    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:54:09.242188 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:09.242144    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:54:09.242594 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:09.242146    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:54:19.242160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:19.242116    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:54:19.242564 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:19.242117    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:54:29.241961 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:29.241921    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:54:29.242411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:29.241921    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:54:39.242290 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:39.242213    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:54:39.242632 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:39.242312    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:54:39.385289 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:39.385260    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:54:39.513999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:39.513898    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container" containerID="cri-o://b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512" gracePeriod=30
Sep 17 17:54:39.513999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:39.513923    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kube-rbac-proxy" containerID="cri-o://b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865" gracePeriod=30
Sep 17 17:54:40.520274 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:40.520241    2575 generic.go:356] "Generic (PLEG): container finished" podID="6041bf58-8ed7-4c98-a125-75578c31effd" containerID="b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865" exitCode=2
Sep 17 17:54:40.520619 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:40.520313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerDied","Data":"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"}
Sep 17 17:54:42.947541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:42.947519    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:54:43.100681 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100657    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location\") pod \"6041bf58-8ed7-4c98-a125-75578c31effd\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") "
Sep 17 17:54:43.100806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100718    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk2jp\" (UniqueName: \"kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp\") pod \"6041bf58-8ed7-4c98-a125-75578c31effd\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") "
Sep 17 17:54:43.100806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100749    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config\") pod \"6041bf58-8ed7-4c98-a125-75578c31effd\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") "
Sep 17 17:54:43.100806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100777    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert\") pod \"6041bf58-8ed7-4c98-a125-75578c31effd\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") "
Sep 17 17:54:43.100806 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100793    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") pod \"6041bf58-8ed7-4c98-a125-75578c31effd\" (UID: \"6041bf58-8ed7-4c98-a125-75578c31effd\") "
Sep 17 17:54:43.101052 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.100881    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6041bf58-8ed7-4c98-a125-75578c31effd" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:43.101052 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.101037    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6041bf58-8ed7-4c98-a125-75578c31effd-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:43.101178 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.101156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6041bf58-8ed7-4c98-a125-75578c31effd" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:43.101178 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.101162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6041bf58-8ed7-4c98-a125-75578c31effd" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:43.102768 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.102745    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6041bf58-8ed7-4c98-a125-75578c31effd" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:43.102857 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.102791    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp" (OuterVolumeSpecName: "kube-api-access-jk2jp") pod "6041bf58-8ed7-4c98-a125-75578c31effd" (UID: "6041bf58-8ed7-4c98-a125-75578c31effd"). InnerVolumeSpecName "kube-api-access-jk2jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:43.201895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.201864    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk2jp\" (UniqueName: \"kubernetes.io/projected/6041bf58-8ed7-4c98-a125-75578c31effd-kube-api-access-jk2jp\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:43.201895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.201891    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:43.201895 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.201901    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6041bf58-8ed7-4c98-a125-75578c31effd-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:43.202060 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.201910    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6041bf58-8ed7-4c98-a125-75578c31effd-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:43.533977 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.533927    2575 generic.go:356] "Generic (PLEG): container finished" podID="6041bf58-8ed7-4c98-a125-75578c31effd" containerID="b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512" exitCode=0
Sep 17 17:54:43.533977 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.533970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerDied","Data":"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"}
Sep 17 17:54:43.534128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.533982    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"
Sep 17 17:54:43.534128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.534015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b" event={"ID":"6041bf58-8ed7-4c98-a125-75578c31effd","Type":"ContainerDied","Data":"35903e7a6584f634b7f71299584e5aaa94ab9c2bdf40a59068393d405d900b4d"}
Sep 17 17:54:43.534128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.534041    2575 scope.go:117] "RemoveContainer" containerID="b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"
Sep 17 17:54:43.542175 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.542157    2575 scope.go:117] "RemoveContainer" containerID="b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"
Sep 17 17:54:43.549113 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.549086    2575 scope.go:117] "RemoveContainer" containerID="ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"
Sep 17 17:54:43.557514 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.557493    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:54:43.561386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561370    2575 scope.go:117] "RemoveContainer" containerID="b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"
Sep 17 17:54:43.561665 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:54:43.561648    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865\": container with ID starting with b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865 not found: ID does not exist" containerID="b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"
Sep 17 17:54:43.561715 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561671    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865"} err="failed to get container status \"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865\": rpc error: code = NotFound desc = could not find container \"b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865\": container with ID starting with b6557b0e267af279a97412ba9f6c7aa96d4c28a56918f43ca20c9f192733e865 not found: ID does not exist"
Sep 17 17:54:43.561715 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561687    2575 scope.go:117] "RemoveContainer" containerID="b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"
Sep 17 17:54:43.561908 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561876    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-l5p9b"]
Sep 17 17:54:43.561988 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:54:43.561918    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512\": container with ID starting with b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512 not found: ID does not exist" containerID="b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"
Sep 17 17:54:43.561988 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561943    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512"} err="failed to get container status \"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512\": rpc error: code = NotFound desc = could not find container \"b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512\": container with ID starting with b322f63037ff363e1be4627e2126bc5a9137eb0b2b9fc59ba1d68783153db512 not found: ID does not exist"
Sep 17 17:54:43.561988 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.561958    2575 scope.go:117] "RemoveContainer" containerID="ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"
Sep 17 17:54:43.562250 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:54:43.562232    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968\": container with ID starting with ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968 not found: ID does not exist" containerID="ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"
Sep 17 17:54:43.562293 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:43.562255    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968"} err="failed to get container status \"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968\": rpc error: code = NotFound desc = could not find container \"ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968\": container with ID starting with ddb837586d1772e1272a45341cb8ad3f38efc3e370c22c14776530484d71d968 not found: ID does not exist"
Sep 17 17:54:44.777146 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:44.777119    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" path="/var/lib/kubelet/pods/6041bf58-8ed7-4c98-a125-75578c31effd/volumes"
Sep 17 17:54:46.755752 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:46.755720    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:54:46.756180 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:46.755954    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" containerID="cri-o://a7be253bf090678cdea5a8ac5f707343bd15ff603c798858ef58bbaf9f4f99d8" gracePeriod=30
Sep 17 17:54:46.756180 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:46.756001    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kube-rbac-proxy" containerID="cri-o://6990877f6d646c371139d8cb2c65fec876fe7723bfec82586345cd97232c953b" gracePeriod=30
Sep 17 17:54:47.552031 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:47.552001    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerID="6990877f6d646c371139d8cb2c65fec876fe7723bfec82586345cd97232c953b" exitCode=2
Sep 17 17:54:47.552220 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:47.552083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerDied","Data":"6990877f6d646c371139d8cb2c65fec876fe7723bfec82586345cd97232c953b"}
Sep 17 17:54:49.237859 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:49.237820    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:54:49.242166 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:49.242142    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 17:54:50.566471 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.566440    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerID="a7be253bf090678cdea5a8ac5f707343bd15ff603c798858ef58bbaf9f4f99d8" exitCode=0
Sep 17 17:54:50.566809 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.566513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerDied","Data":"a7be253bf090678cdea5a8ac5f707343bd15ff603c798858ef58bbaf9f4f99d8"}
Sep 17 17:54:50.591684 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.591660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:54:50.650747 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.650721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls\") pod \"b2854300-a64d-430c-9d4c-0f1d5f32331d\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") "
Sep 17 17:54:50.650854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.650777    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config\") pod \"b2854300-a64d-430c-9d4c-0f1d5f32331d\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") "
Sep 17 17:54:50.650854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.650801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location\") pod \"b2854300-a64d-430c-9d4c-0f1d5f32331d\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") "
Sep 17 17:54:50.650854 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.650825    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert\") pod \"b2854300-a64d-430c-9d4c-0f1d5f32331d\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") "
Sep 17 17:54:50.650970 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.650855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgsnx\" (UniqueName: \"kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx\") pod \"b2854300-a64d-430c-9d4c-0f1d5f32331d\" (UID: \"b2854300-a64d-430c-9d4c-0f1d5f32331d\") "
Sep 17 17:54:50.651189 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.651156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2854300-a64d-430c-9d4c-0f1d5f32331d" (UID: "b2854300-a64d-430c-9d4c-0f1d5f32331d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:50.651301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.651193    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2854300-a64d-430c-9d4c-0f1d5f32331d" (UID: "b2854300-a64d-430c-9d4c-0f1d5f32331d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:50.651301 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.651188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2854300-a64d-430c-9d4c-0f1d5f32331d" (UID: "b2854300-a64d-430c-9d4c-0f1d5f32331d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:50.652792 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.652775    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2854300-a64d-430c-9d4c-0f1d5f32331d" (UID: "b2854300-a64d-430c-9d4c-0f1d5f32331d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:50.652878 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.652830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx" (OuterVolumeSpecName: "kube-api-access-mgsnx") pod "b2854300-a64d-430c-9d4c-0f1d5f32331d" (UID: "b2854300-a64d-430c-9d4c-0f1d5f32331d"). InnerVolumeSpecName "kube-api-access-mgsnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:50.751778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.751716    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:50.751778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.751740    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsnx\" (UniqueName: \"kubernetes.io/projected/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-api-access-mgsnx\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:50.751778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.751751    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2854300-a64d-430c-9d4c-0f1d5f32331d-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:50.751778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.751761    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2854300-a64d-430c-9d4c-0f1d5f32331d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:50.751778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:50.751769    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2854300-a64d-430c-9d4c-0f1d5f32331d-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.572087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.572056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq" event={"ID":"b2854300-a64d-430c-9d4c-0f1d5f32331d","Type":"ContainerDied","Data":"c97912df32d66048a16088434c726b3e7aa4781b8cee0be59b0f2f5a0b430c36"}
Sep 17 17:54:51.572479 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.572122    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"
Sep 17 17:54:51.572479 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.572141    2575 scope.go:117] "RemoveContainer" containerID="6990877f6d646c371139d8cb2c65fec876fe7723bfec82586345cd97232c953b"
Sep 17 17:54:51.579472 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.579455    2575 scope.go:117] "RemoveContainer" containerID="a7be253bf090678cdea5a8ac5f707343bd15ff603c798858ef58bbaf9f4f99d8"
Sep 17 17:54:51.586136 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.586118    2575 scope.go:117] "RemoveContainer" containerID="31fd3e47e6bb65882bfef4592c0015632a7c83110d18c658f5a3471fb4aa3527"
Sep 17 17:54:51.595023 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.595002    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:54:51.597298 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.597277    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-66ppq"]
Sep 17 17:54:51.925058 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925031    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:54:51.925297 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925284    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925299    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925307    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925313    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925324    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925337    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925342    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925352    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925357    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925366    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925371    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925379    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925384    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925390    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925395    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925402    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925407    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="storage-initializer"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925444    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925452    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6041bf58-8ed7-4c98-a125-75578c31effd" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925458    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kube-rbac-proxy"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925464    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44a0410-ec3b-4e3e-99b1-f91b34ca3012" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925470    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kserve-container"
Sep 17 17:54:51.925551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.925477    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" containerName="kube-rbac-proxy"
Sep 17 17:54:51.929818 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.929788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:51.932145 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.932120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:54:51.932544 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.932292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 17:54:51.932544 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.932322    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 17:54:51.932544 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.932359    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:54:51.932795 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.932646    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:54:51.933322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.933306    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:54:51.933411 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.933380    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:54:51.941217 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.941196    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:54:51.998135 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:51.998111    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:54:52.001334 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.001319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.003726 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.003698    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 17:54:52.012350 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.012330    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:54:52.059809 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7q85\" (UniqueName: \"kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.059914 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.059914 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.059914 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.060040 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.060040 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.059978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.060040 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.060017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.060184 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.060051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.060184 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.060080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kd9\" (UniqueName: \"kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.060184 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.060124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161198 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161325 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161325 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.161325 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.161325 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kd9\" (UniqueName: \"kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.161551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q85\" (UniqueName: \"kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.161551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.161551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161663    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.161798 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.161673    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.162083 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.162057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.162484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.162465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.162551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.162525    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.162551 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.162535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.163836 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.163819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.163908 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.163892    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.171348 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.171324    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q85\" (UniqueName: \"kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-tms4h\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.171597 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.171576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kd9\" (UniqueName: \"kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-4t767\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.241517 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.241450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:52.310607 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.310576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:52.517430 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.517402    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:54:52.519160 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:54:52.519131    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4942059e_9920_4057_966a_bdc4e7ef726b.slice/crio-370be79f84ef0a6660a30b60c98cadbfdb8c6995669abdbf04a392096b491b3c WatchSource:0}: Error finding container 370be79f84ef0a6660a30b60c98cadbfdb8c6995669abdbf04a392096b491b3c: Status 404 returned error can't find the container with id 370be79f84ef0a6660a30b60c98cadbfdb8c6995669abdbf04a392096b491b3c
Sep 17 17:54:52.578878 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.578845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerStarted","Data":"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"}
Sep 17 17:54:52.579246 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.579157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerStarted","Data":"370be79f84ef0a6660a30b60c98cadbfdb8c6995669abdbf04a392096b491b3c"}
Sep 17 17:54:52.580972 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.580575    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:54:52.777155 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:52.777067    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2854300-a64d-430c-9d4c-0f1d5f32331d" path="/var/lib/kubelet/pods/b2854300-a64d-430c-9d4c-0f1d5f32331d/volumes"
Sep 17 17:54:53.584758 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:53.584715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerStarted","Data":"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"}
Sep 17 17:54:53.585178 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:53.584788    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerStarted","Data":"c2d62895b56ccc6bb10221f66cf2a72875fa1d41ed0cfb919b3e5bb3c42648f1"}
Sep 17 17:54:56.596768 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:56.596742    2575 generic.go:356] "Generic (PLEG): container finished" podID="14ef5408-0963-4831-88fe-e2e39ade5941" containerID="24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f" exitCode=0
Sep 17 17:54:56.597087 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:56.596817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerDied","Data":"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"}
Sep 17 17:54:56.598395 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:56.598374    2575 generic.go:356] "Generic (PLEG): container finished" podID="4942059e-9920-4057-966a-bdc4e7ef726b" containerID="aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c" exitCode=0
Sep 17 17:54:56.598486 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:56.598417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerDied","Data":"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"}
Sep 17 17:54:57.604587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.604548    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerStarted","Data":"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"}
Sep 17 17:54:57.605048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.604648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerStarted","Data":"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"}
Sep 17 17:54:57.605048 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.604919    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:57.606766 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.606745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerStarted","Data":"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"}
Sep 17 17:54:57.606848 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.606793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerStarted","Data":"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"}
Sep 17 17:54:57.607121 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.607090    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:57.607160 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.607154    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:54:57.608450 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.608425    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:54:57.626686 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.626633    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podStartSLOduration=2.801326801 podStartE2EDuration="6.626616541s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82528974s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:57.624762802 +0000 UTC m=+1268.991894684" watchObservedRunningTime="2026-09-17 17:54:57.626616541 +0000 UTC m=+1268.993748424"
Sep 17 17:54:57.646278 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:57.646235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podStartSLOduration=2.851239325 podStartE2EDuration="6.646221931s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794982606s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:57.645162567 +0000 UTC m=+1269.012294488" watchObservedRunningTime="2026-09-17 17:54:57.646221931 +0000 UTC m=+1269.013353811"
Sep 17 17:54:58.611138 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:58.611076    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:54:58.613218 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:58.611351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:54:58.613218 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:58.612446    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:54:59.614108 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:54:59.614068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:03.615280 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:03.615253    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:55:03.615785 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:03.615754    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:04.618949 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:04.618919    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:55:04.619449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:04.619424    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:13.616548 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:13.616507    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:14.619500 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:14.619456    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:23.616158 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:23.616091    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:24.619455 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:24.619419    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:33.616349 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:33.616310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:34.620230 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:34.620186    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:43.615742 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:43.615695    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:44.619898 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:44.619854    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:55:53.615778 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:53.615736    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:54.620277 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:55:54.620235    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:56:03.616342 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:03.616244    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:56:04.620017 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:04.619978    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:56:13.616780 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:13.616741    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:56:14.620291 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.620256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:56:14.763919 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.763890    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:56:14.764167 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.764080    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container" containerID="cri-o://bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f" gracePeriod=30
Sep 17 17:56:14.764167 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.764131    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kube-rbac-proxy" containerID="cri-o://e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7" gracePeriod=30
Sep 17 17:56:14.887931 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.887903    2575 generic.go:356] "Generic (PLEG): container finished" podID="14ef5408-0963-4831-88fe-e2e39ade5941" containerID="e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7" exitCode=2
Sep 17 17:56:14.888036 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:14.887949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerDied","Data":"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"}
Sep 17 17:56:18.497199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.497176    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:56:18.580556 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580531    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert\") pod \"14ef5408-0963-4831-88fe-e2e39ade5941\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") "
Sep 17 17:56:18.580699 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580575    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config\") pod \"14ef5408-0963-4831-88fe-e2e39ade5941\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") "
Sep 17 17:56:18.580699 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580612    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls\") pod \"14ef5408-0963-4831-88fe-e2e39ade5941\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") "
Sep 17 17:56:18.580699 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580652    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location\") pod \"14ef5408-0963-4831-88fe-e2e39ade5941\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") "
Sep 17 17:56:18.580699 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580676    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9kd9\" (UniqueName: \"kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9\") pod \"14ef5408-0963-4831-88fe-e2e39ade5941\" (UID: \"14ef5408-0963-4831-88fe-e2e39ade5941\") "
Sep 17 17:56:18.580949 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580924    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14ef5408-0963-4831-88fe-e2e39ade5941" (UID: "14ef5408-0963-4831-88fe-e2e39ade5941"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:18.580996 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.580972    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14ef5408-0963-4831-88fe-e2e39ade5941" (UID: "14ef5408-0963-4831-88fe-e2e39ade5941"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:18.581042 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.581020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14ef5408-0963-4831-88fe-e2e39ade5941" (UID: "14ef5408-0963-4831-88fe-e2e39ade5941"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:18.582679 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.582658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14ef5408-0963-4831-88fe-e2e39ade5941" (UID: "14ef5408-0963-4831-88fe-e2e39ade5941"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:18.582790 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.582768    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9" (OuterVolumeSpecName: "kube-api-access-v9kd9") pod "14ef5408-0963-4831-88fe-e2e39ade5941" (UID: "14ef5408-0963-4831-88fe-e2e39ade5941"). InnerVolumeSpecName "kube-api-access-v9kd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:18.681286 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.681263    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:56:18.681286 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.681284    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ef5408-0963-4831-88fe-e2e39ade5941-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:56:18.681442 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.681294    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ef5408-0963-4831-88fe-e2e39ade5941-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:56:18.681442 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.681304    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14ef5408-0963-4831-88fe-e2e39ade5941-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:56:18.681442 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.681312    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9kd9\" (UniqueName: \"kubernetes.io/projected/14ef5408-0963-4831-88fe-e2e39ade5941-kube-api-access-v9kd9\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:56:18.906250 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.906193    2575 generic.go:356] "Generic (PLEG): container finished" podID="14ef5408-0963-4831-88fe-e2e39ade5941" containerID="bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f" exitCode=0
Sep 17 17:56:18.906338 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.906248    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"
Sep 17 17:56:18.906338 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.906264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerDied","Data":"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"}
Sep 17 17:56:18.906338 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.906316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767" event={"ID":"14ef5408-0963-4831-88fe-e2e39ade5941","Type":"ContainerDied","Data":"c2d62895b56ccc6bb10221f66cf2a72875fa1d41ed0cfb919b3e5bb3c42648f1"}
Sep 17 17:56:18.906439 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.906344    2575 scope.go:117] "RemoveContainer" containerID="e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"
Sep 17 17:56:18.913999 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.913982    2575 scope.go:117] "RemoveContainer" containerID="bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"
Sep 17 17:56:18.920872 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.920857    2575 scope.go:117] "RemoveContainer" containerID="24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"
Sep 17 17:56:18.927138 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.927116    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:56:18.927849 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.927832    2575 scope.go:117] "RemoveContainer" containerID="e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"
Sep 17 17:56:18.928128 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:56:18.928084    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7\": container with ID starting with e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7 not found: ID does not exist" containerID="e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"
Sep 17 17:56:18.928222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.928141    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7"} err="failed to get container status \"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7\": rpc error: code = NotFound desc = could not find container \"e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7\": container with ID starting with e329e35777b77dc3ef0340adea20beb9a5fd69316c2f3af8ac81e27de8b303b7 not found: ID does not exist"
Sep 17 17:56:18.928222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.928164    2575 scope.go:117] "RemoveContainer" containerID="bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"
Sep 17 17:56:18.928455 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:56:18.928436    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f\": container with ID starting with bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f not found: ID does not exist" containerID="bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"
Sep 17 17:56:18.928497 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.928462    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f"} err="failed to get container status \"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f\": rpc error: code = NotFound desc = could not find container \"bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f\": container with ID starting with bda45c100b0f68f0de34a74cffa02eb45a20fd532bd46aa125d62f85683d3a8f not found: ID does not exist"
Sep 17 17:56:18.928497 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.928477    2575 scope.go:117] "RemoveContainer" containerID="24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"
Sep 17 17:56:18.928739 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:56:18.928722    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f\": container with ID starting with 24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f not found: ID does not exist" containerID="24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"
Sep 17 17:56:18.928782 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.928744    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f"} err="failed to get container status \"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f\": rpc error: code = NotFound desc = could not find container \"24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f\": container with ID starting with 24a967e4402d11950086133e277518f5f63663ead14f220a77e6e4046561d19f not found: ID does not exist"
Sep 17 17:56:18.931722 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:18.931703    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-4t767"]
Sep 17 17:56:20.779149 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:56:20.779090    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" path="/var/lib/kubelet/pods/14ef5408-0963-4831-88fe-e2e39ade5941/volumes"
Sep 17 17:57:52.228107 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:52.228017    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:57:52.228587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:52.228277    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" containerID="cri-o://a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433" gracePeriod=30
Sep 17 17:57:52.228587 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:52.228324    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kube-rbac-proxy" containerID="cri-o://ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70" gracePeriod=30
Sep 17 17:57:53.250049 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:53.250016    2575 generic.go:356] "Generic (PLEG): container finished" podID="4942059e-9920-4057-966a-bdc4e7ef726b" containerID="ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70" exitCode=2
Sep 17 17:57:53.250049 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:53.250053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerDied","Data":"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"}
Sep 17 17:57:54.614678 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:54.614639    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 17:57:54.619984 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:54.619960    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:57:56.074344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.074322    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:57:56.136199 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config\") pod \"4942059e-9920-4057-966a-bdc4e7ef726b\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") "
Sep 17 17:57:56.136323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136247    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls\") pod \"4942059e-9920-4057-966a-bdc4e7ef726b\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") "
Sep 17 17:57:56.136323 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136280    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location\") pod \"4942059e-9920-4057-966a-bdc4e7ef726b\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") "
Sep 17 17:57:56.136440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136322    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert\") pod \"4942059e-9920-4057-966a-bdc4e7ef726b\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") "
Sep 17 17:57:56.136440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136355    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4942059e-9920-4057-966a-bdc4e7ef726b" (UID: "4942059e-9920-4057-966a-bdc4e7ef726b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:56.136440 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136364    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7q85\" (UniqueName: \"kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85\") pod \"4942059e-9920-4057-966a-bdc4e7ef726b\" (UID: \"4942059e-9920-4057-966a-bdc4e7ef726b\") "
Sep 17 17:57:56.136581 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136560    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.136618 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136590    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4942059e-9920-4057-966a-bdc4e7ef726b" (UID: "4942059e-9920-4057-966a-bdc4e7ef726b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:56.136657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.136620    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4942059e-9920-4057-966a-bdc4e7ef726b" (UID: "4942059e-9920-4057-966a-bdc4e7ef726b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:56.138322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.138297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85" (OuterVolumeSpecName: "kube-api-access-f7q85") pod "4942059e-9920-4057-966a-bdc4e7ef726b" (UID: "4942059e-9920-4057-966a-bdc4e7ef726b"). InnerVolumeSpecName "kube-api-access-f7q85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:56.138322 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.138302    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4942059e-9920-4057-966a-bdc4e7ef726b" (UID: "4942059e-9920-4057-966a-bdc4e7ef726b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:56.236889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.236824    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7q85\" (UniqueName: \"kubernetes.io/projected/4942059e-9920-4057-966a-bdc4e7ef726b-kube-api-access-f7q85\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.236889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.236855    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4942059e-9920-4057-966a-bdc4e7ef726b-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.236889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.236867    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4942059e-9920-4057-966a-bdc4e7ef726b-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.236889 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.236875    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4942059e-9920-4057-966a-bdc4e7ef726b-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.262539 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.262516    2575 generic.go:356] "Generic (PLEG): container finished" podID="4942059e-9920-4057-966a-bdc4e7ef726b" containerID="a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433" exitCode=0
Sep 17 17:57:56.262657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.262589    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"
Sep 17 17:57:56.262657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.262597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerDied","Data":"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"}
Sep 17 17:57:56.262657 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.262647    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h" event={"ID":"4942059e-9920-4057-966a-bdc4e7ef726b","Type":"ContainerDied","Data":"370be79f84ef0a6660a30b60c98cadbfdb8c6995669abdbf04a392096b491b3c"}
Sep 17 17:57:56.262830 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.262676    2575 scope.go:117] "RemoveContainer" containerID="ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"
Sep 17 17:57:56.270815 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.270656    2575 scope.go:117] "RemoveContainer" containerID="a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"
Sep 17 17:57:56.278195 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.278176    2575 scope.go:117] "RemoveContainer" containerID="aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"
Sep 17 17:57:56.285765 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.285739    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:57:56.286112 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286082    2575 scope.go:117] "RemoveContainer" containerID="ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"
Sep 17 17:57:56.286401 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:57:56.286380    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70\": container with ID starting with ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70 not found: ID does not exist" containerID="ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"
Sep 17 17:57:56.286449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286411    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70"} err="failed to get container status \"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70\": rpc error: code = NotFound desc = could not find container \"ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70\": container with ID starting with ad2d6a23d742874c0a3a40cef86b2dad61de4229d1bf9c13696d1d8142929b70 not found: ID does not exist"
Sep 17 17:57:56.286449 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286429    2575 scope.go:117] "RemoveContainer" containerID="a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"
Sep 17 17:57:56.286697 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:57:56.286681    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433\": container with ID starting with a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433 not found: ID does not exist" containerID="a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"
Sep 17 17:57:56.286757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286699    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433"} err="failed to get container status \"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433\": rpc error: code = NotFound desc = could not find container \"a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433\": container with ID starting with a94494a9223160413c03ff491ed78a24786585c5bc4a336fb18aa7a9d0f7b433 not found: ID does not exist"
Sep 17 17:57:56.286757 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286711    2575 scope.go:117] "RemoveContainer" containerID="aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"
Sep 17 17:57:56.286903 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:57:56.286888    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c\": container with ID starting with aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c not found: ID does not exist" containerID="aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"
Sep 17 17:57:56.286942 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.286906    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c"} err="failed to get container status \"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c\": rpc error: code = NotFound desc = could not find container \"aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c\": container with ID starting with aab588b89b39d313788179ea999b8cf427dc40ebd150d5520aa525112ba8f22c not found: ID does not exist"
Sep 17 17:57:56.289557 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.289535    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-tms4h"]
Sep 17 17:57:56.777214 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:57:56.777187    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" path="/var/lib/kubelet/pods/4942059e-9920-4057-966a-bdc4e7ef726b/volumes"
Sep 17 17:58:06.527975 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.527945    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528196    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528209    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528223    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="storage-initializer"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528232    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="storage-initializer"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528245    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528251    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528260    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528265    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528274    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="storage-initializer"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528279    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="storage-initializer"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528286    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528291    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528331    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528339    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kube-rbac-proxy"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528345    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ef5408-0963-4831-88fe-e2e39ade5941" containerName="kserve-container"
Sep 17 17:58:06.528371 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.528351    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4942059e-9920-4057-966a-bdc4e7ef726b" containerName="kserve-container"
Sep 17 17:58:06.532629 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.532606    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.535276 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.535243    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 17:58:06.535399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.535339    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:58:06.535399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.535250    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 17:58:06.535518 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.535470    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:58:06.536306 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.536288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:58:06.540389 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.540368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:58:06.603261 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.603241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.603335 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.603279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wxd\" (UniqueName: \"kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.603335 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.603303    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.704140 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.704116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wxd\" (UniqueName: \"kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.704236 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.704151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.704236 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.704205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.704795 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.704777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.706541 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.706521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.712810 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.712792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wxd\" (UniqueName: \"kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-pgg8v\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:06.843424 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:06.843396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:07.110615 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.110470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:58:07.113286 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:58:07.113253    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0da48c8_a108_49ed_9ede_8adfe3860bc8.slice/crio-e5732bcad1dcd5f9d7423526e12370496554bf49e307c0e71ceae18288aff02f WatchSource:0}: Error finding container e5732bcad1dcd5f9d7423526e12370496554bf49e307c0e71ceae18288aff02f: Status 404 returned error can't find the container with id e5732bcad1dcd5f9d7423526e12370496554bf49e307c0e71ceae18288aff02f
Sep 17 17:58:07.307174 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.307135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerStarted","Data":"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"}
Sep 17 17:58:07.307285 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.307226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerStarted","Data":"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"}
Sep 17 17:58:07.307401 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.307366    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:07.307502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.307425    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:07.307502 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.307447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerStarted","Data":"e5732bcad1dcd5f9d7423526e12370496554bf49e307c0e71ceae18288aff02f"}
Sep 17 17:58:07.308933 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.308915    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:07.324909 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:07.324846    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" podStartSLOduration=1.324828768 podStartE2EDuration="1.324828768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:07.324402829 +0000 UTC m=+1458.691534711" watchObservedRunningTime="2026-09-17 17:58:07.324828768 +0000 UTC m=+1458.691960650"
Sep 17 17:58:14.321858 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:14.321814    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:58:16.565768 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.565731    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 17:58:16.569115 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.569077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.571428 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.571408    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:58:16.571533 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.571415    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 17:58:16.571600 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.571536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:58:16.571707 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.571692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 17:58:16.579903 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.579885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 17:58:16.675927 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.675900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvl7\" (UniqueName: \"kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.676021 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.675944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.676021 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.675973    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.676128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.676049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.676128 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.676084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776555 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.776531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.776568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.776586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.776613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvl7\" (UniqueName: \"kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776661 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.776653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.776896 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:58:16.776712    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 17 17:58:16.776896 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:58:16.776791    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls podName:de0bbf09-3f87-4462-afdb-4b98ab34c72d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.276769006 +0000 UTC m=+1468.643900873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls") pod "isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 17 17:58:16.777115 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.777076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.777256 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.777238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.777256 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.777250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:16.788256 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:16.788236    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvl7\" (UniqueName: \"kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:17.280302 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:17.280270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:17.282809 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:17.282782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b854445b6-qshtl\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:17.479222 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:17.479191    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:17.756529 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:17.756440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 17:58:17.759149 ip-10-0-134-52 kubenswrapper[2575]: W0917 17:58:17.759110    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0bbf09_3f87_4462_afdb_4b98ab34c72d.slice/crio-8e083425913c42a2a4fa9e38ad5240377744f71fd8af8a3f50497ee845550da4 WatchSource:0}: Error finding container 8e083425913c42a2a4fa9e38ad5240377744f71fd8af8a3f50497ee845550da4: Status 404 returned error can't find the container with id 8e083425913c42a2a4fa9e38ad5240377744f71fd8af8a3f50497ee845550da4
Sep 17 17:58:18.352519 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:18.352484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerStarted","Data":"5e3a07aa181013b05f0eca9f1fa17714cdf3b4b04e1e8ef9deaa9b5ce17a9f91"}
Sep 17 17:58:18.352672 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:18.352545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerStarted","Data":"8e083425913c42a2a4fa9e38ad5240377744f71fd8af8a3f50497ee845550da4"}
Sep 17 17:58:22.368249 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:22.368216    2575 generic.go:356] "Generic (PLEG): container finished" podID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerID="5e3a07aa181013b05f0eca9f1fa17714cdf3b4b04e1e8ef9deaa9b5ce17a9f91" exitCode=0
Sep 17 17:58:22.368612 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:22.368290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerDied","Data":"5e3a07aa181013b05f0eca9f1fa17714cdf3b4b04e1e8ef9deaa9b5ce17a9f91"}
Sep 17 17:58:23.373898 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.373863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerStarted","Data":"dddac0f5f372481943e081e4d885dc15921550eb6581affce5baba33d5000753"}
Sep 17 17:58:23.374356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.373928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerStarted","Data":"d3ee4355bbb7a794a15ca78034183a3c48c0696682acd500ee39a099d8e39e16"}
Sep 17 17:58:23.374356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.373953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerStarted","Data":"cdc10594ce8d7c1171fbff9207f4d8dd92b5d8054a2c1b75e7239a4ee8d87591"}
Sep 17 17:58:23.374356 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.374232    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:23.374512 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.374390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:23.375595 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.375572    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:23.396158 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:23.396078    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podStartSLOduration=3.5526479909999997 podStartE2EDuration="7.396065644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.843417653s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:23.3949167 +0000 UTC m=+1474.762048593" watchObservedRunningTime="2026-09-17 17:58:23.396065644 +0000 UTC m=+1474.763197525"
Sep 17 17:58:24.384390 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:24.384352    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:24.384390 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:24.384360    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:24.385420 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:24.385395    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:25.388311 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:25.388269    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:25.388781 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:25.388756    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:30.392868 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:30.392834    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:58:30.393406 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:30.393373    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:30.393625 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:30.393599    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:40.393691 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:40.393642    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:40.394082 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:40.394044    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:50.393365 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:50.393319    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:58:50.393829 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:50.393805    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:50.785943 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:50.785877    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:58:50.788480 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:58:50.788457    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 17:59:00.393890 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:00.393851    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:59:00.394429 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:00.394404    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:10.394203 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:10.394163    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:59:10.394692 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:10.394667    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:20.394044 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:20.393997    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:59:20.394484 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:20.394460    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:30.393503 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:30.393459    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:59:30.393934 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:30.393911    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:40.394847 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:40.394815    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:59:40.395399 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:40.395056    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 17:59:51.586519 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.586470    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-pgg8v_e0da48c8-a108-49ed-9ede-8adfe3860bc8/kserve-container/0.log"
Sep 17 17:59:51.774125 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.774085    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 17:59:51.774397 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.774368    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" containerID="cri-o://cdc10594ce8d7c1171fbff9207f4d8dd92b5d8054a2c1b75e7239a4ee8d87591" gracePeriod=30
Sep 17 17:59:51.774510 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.774389    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" containerID="cri-o://dddac0f5f372481943e081e4d885dc15921550eb6581affce5baba33d5000753" gracePeriod=30
Sep 17 17:59:51.774510 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.774390    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" containerID="cri-o://d3ee4355bbb7a794a15ca78034183a3c48c0696682acd500ee39a099d8e39e16" gracePeriod=30
Sep 17 17:59:51.821162 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.821132    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:59:51.821369 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.821341    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kserve-container" containerID="cri-o://400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50" gracePeriod=30
Sep 17 17:59:51.821505 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:51.821383    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kube-rbac-proxy" containerID="cri-o://3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669" gracePeriod=30
Sep 17 17:59:52.059813 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.059791    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:59:52.107344 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.107317    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config\") pod \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") "
Sep 17 17:59:52.107447 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.107357    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls\") pod \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") "
Sep 17 17:59:52.107447 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.107380    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4wxd\" (UniqueName: \"kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd\") pod \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\" (UID: \"e0da48c8-a108-49ed-9ede-8adfe3860bc8\") "
Sep 17 17:59:52.107685 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.107665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0da48c8-a108-49ed-9ede-8adfe3860bc8" (UID: "e0da48c8-a108-49ed-9ede-8adfe3860bc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:52.109314 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.109291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0da48c8-a108-49ed-9ede-8adfe3860bc8" (UID: "e0da48c8-a108-49ed-9ede-8adfe3860bc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:52.109382 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.109343    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd" (OuterVolumeSpecName: "kube-api-access-l4wxd") pod "e0da48c8-a108-49ed-9ede-8adfe3860bc8" (UID: "e0da48c8-a108-49ed-9ede-8adfe3860bc8"). InnerVolumeSpecName "kube-api-access-l4wxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:52.208386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.208335    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:59:52.208386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.208357    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0da48c8-a108-49ed-9ede-8adfe3860bc8-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:59:52.208386 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.208367    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wxd\" (UniqueName: \"kubernetes.io/projected/e0da48c8-a108-49ed-9ede-8adfe3860bc8-kube-api-access-l4wxd\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 17:59:52.707577 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.707542    2575 generic.go:356] "Generic (PLEG): container finished" podID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerID="d3ee4355bbb7a794a15ca78034183a3c48c0696682acd500ee39a099d8e39e16" exitCode=2
Sep 17 17:59:52.707992 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.707612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerDied","Data":"d3ee4355bbb7a794a15ca78034183a3c48c0696682acd500ee39a099d8e39e16"}
Sep 17 17:59:52.709171 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709149    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerID="3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669" exitCode=2
Sep 17 17:59:52.709171 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709167    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerID="400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50" exitCode=2
Sep 17 17:59:52.709319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerDied","Data":"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"}
Sep 17 17:59:52.709319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709195    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"
Sep 17 17:59:52.709319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerDied","Data":"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"}
Sep 17 17:59:52.709319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v" event={"ID":"e0da48c8-a108-49ed-9ede-8adfe3860bc8","Type":"ContainerDied","Data":"e5732bcad1dcd5f9d7423526e12370496554bf49e307c0e71ceae18288aff02f"}
Sep 17 17:59:52.709319 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.709274    2575 scope.go:117] "RemoveContainer" containerID="3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"
Sep 17 17:59:52.717531 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.717512    2575 scope.go:117] "RemoveContainer" containerID="400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"
Sep 17 17:59:52.724666 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.724649    2575 scope.go:117] "RemoveContainer" containerID="3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"
Sep 17 17:59:52.724969 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:59:52.724949    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669\": container with ID starting with 3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669 not found: ID does not exist" containerID="3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"
Sep 17 17:59:52.725018 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.724978    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"} err="failed to get container status \"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669\": rpc error: code = NotFound desc = could not find container \"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669\": container with ID starting with 3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669 not found: ID does not exist"
Sep 17 17:59:52.725018 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.724995    2575 scope.go:117] "RemoveContainer" containerID="400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"
Sep 17 17:59:52.725258 ip-10-0-134-52 kubenswrapper[2575]: E0917 17:59:52.725240    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50\": container with ID starting with 400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50 not found: ID does not exist" containerID="400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"
Sep 17 17:59:52.725332 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.725268    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"} err="failed to get container status \"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50\": rpc error: code = NotFound desc = could not find container \"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50\": container with ID starting with 400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50 not found: ID does not exist"
Sep 17 17:59:52.725332 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.725290    2575 scope.go:117] "RemoveContainer" containerID="3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"
Sep 17 17:59:52.725542 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.725523    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669"} err="failed to get container status \"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669\": rpc error: code = NotFound desc = could not find container \"3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669\": container with ID starting with 3628dd5da53bc36a3ec4edd5391b2f95eb8d3f38688c0ab245aba4295f4be669 not found: ID does not exist"
Sep 17 17:59:52.725592 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.725543    2575 scope.go:117] "RemoveContainer" containerID="400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"
Sep 17 17:59:52.725752 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.725736    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50"} err="failed to get container status \"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50\": rpc error: code = NotFound desc = could not find container \"400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50\": container with ID starting with 400330c458b05068389356c91fcacc1e744e2bace03594229b438ef7e5086b50 not found: ID does not exist"
Sep 17 17:59:52.731325 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.731304    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:59:52.734009 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.733984    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-pgg8v"]
Sep 17 17:59:52.777419 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:52.777398    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" path="/var/lib/kubelet/pods/e0da48c8-a108-49ed-9ede-8adfe3860bc8/volumes"
Sep 17 17:59:55.389267 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:55.389224    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 17:59:55.724793 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:55.724767    2575 generic.go:356] "Generic (PLEG): container finished" podID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerID="cdc10594ce8d7c1171fbff9207f4d8dd92b5d8054a2c1b75e7239a4ee8d87591" exitCode=0
Sep 17 17:59:55.724883 ip-10-0-134-52 kubenswrapper[2575]: I0917 17:59:55.724808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerDied","Data":"cdc10594ce8d7c1171fbff9207f4d8dd92b5d8054a2c1b75e7239a4ee8d87591"}
Sep 17 18:00:00.388420 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:00.388373    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:00:00.393742 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:00.393714    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:00:00.394134 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:00.394078    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:00:05.389080 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:05.389031    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:00:05.389564 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:05.389228    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 18:00:07.625127 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:07.625078    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:07.652360 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:07.652323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:07.664528 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:07.664505    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:07.679045 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:07.679017    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:08.524782 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:08.524755    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:08.539602 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:08.539576    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:08.557477 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:08.557453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:08.576922 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:08.576901    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:09.429771 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:09.429744    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:09.449820 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:09.449792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:09.465551 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:09.465530    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:09.482073 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:09.482036    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:10.287140 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.287109    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:10.301993 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.301962    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:10.317560 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.317534    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:10.336905 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.336882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:10.389169 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.389139    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:00:10.393459 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.393438    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:00:10.393844 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:10.393810    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:00:11.159584 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:11.159563    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:11.173454 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:11.173421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:11.185557 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:11.185534    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:11.200031 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:11.200012    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:12.009771 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.009741    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:12.028591 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.028564    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:12.046165 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.046123    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:12.053330 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.053308    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:12.902640 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.902618    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:12.921210 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.921121    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:12.935579 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.935555    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:12.950966 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:12.950938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:13.766064 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:13.766034    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:13.779555 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:13.779527    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:13.791204 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:13.791170    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:13.803914 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:13.803892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:14.637106 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:14.637073    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:14.656726 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:14.656683    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:14.672762 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:14.672738    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:14.685557 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:14.685529    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:15.388688 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:15.388648    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:00:15.492712 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:15.492673    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:15.507238 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:15.507202    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:15.519316 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:15.519295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:15.533079 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:15.533052    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:16.343730 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:16.343706    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:16.359966 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:16.359936    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:16.372038 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:16.372013    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:16.383553 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:16.383527    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:17.200845 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:17.200809    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:17.220108 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:17.220069    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:17.235374 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:17.235352    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:17.248124 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:17.248090    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:18.096195 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:18.096150    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:18.115638 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:18.115610    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:18.127385 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:18.127363    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:18.145229 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:18.145206    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:19.006038 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.006008    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:19.022255 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.022224    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:19.038360 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.038335    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:19.053869 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.053846    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:19.903328 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.903304    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kserve-container/0.log"
Sep 17 18:00:19.917603 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.917577    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/kube-rbac-proxy/0.log"
Sep 17 18:00:19.930888 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.930856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/agent/0.log"
Sep 17 18:00:19.942981 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:19.942950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b854445b6-qshtl_de0bbf09-3f87-4462-afdb-4b98ab34c72d/storage-initializer/0.log"
Sep 17 18:00:20.389389 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:20.389352    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:00:20.393677 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:20.393646    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:00:20.393830 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:20.393804    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 18:00:20.394091 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:20.394063    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:00:20.394231 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:20.394216    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 18:00:21.827934 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.827906    2575 generic.go:356] "Generic (PLEG): container finished" podID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerID="dddac0f5f372481943e081e4d885dc15921550eb6581affce5baba33d5000753" exitCode=0
Sep 17 18:00:21.828317 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.827978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerDied","Data":"dddac0f5f372481943e081e4d885dc15921550eb6581affce5baba33d5000753"}
Sep 17 18:00:21.921789 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.921768    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 18:00:21.992521 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992496    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config\") pod \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") "
Sep 17 18:00:21.992631 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992543    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location\") pod \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") "
Sep 17 18:00:21.992631 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992572    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert\") pod \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") "
Sep 17 18:00:21.992631 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992622    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsvl7\" (UniqueName: \"kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7\") pod \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") "
Sep 17 18:00:21.992802 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992647    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") pod \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\" (UID: \"de0bbf09-3f87-4462-afdb-4b98ab34c72d\") "
Sep 17 18:00:21.992901 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de0bbf09-3f87-4462-afdb-4b98ab34c72d" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:21.992963 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992948    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de0bbf09-3f87-4462-afdb-4b98ab34c72d" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:21.993022 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.992977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de0bbf09-3f87-4462-afdb-4b98ab34c72d" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:21.994724 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.994694    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7" (OuterVolumeSpecName: "kube-api-access-xsvl7") pod "de0bbf09-3f87-4462-afdb-4b98ab34c72d" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d"). InnerVolumeSpecName "kube-api-access-xsvl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:21.994811 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:21.994721    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de0bbf09-3f87-4462-afdb-4b98ab34c72d" (UID: "de0bbf09-3f87-4462-afdb-4b98ab34c72d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:22.093875 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.093856    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kserve-provision-location\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 18:00:22.093960 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.093876    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-cabundle-cert\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 18:00:22.093960 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.093888    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvl7\" (UniqueName: \"kubernetes.io/projected/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-api-access-xsvl7\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 18:00:22.093960 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.093896    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de0bbf09-3f87-4462-afdb-4b98ab34c72d-proxy-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 18:00:22.093960 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.093906    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de0bbf09-3f87-4462-afdb-4b98ab34c72d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\""
Sep 17 18:00:22.834238 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.834200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl" event={"ID":"de0bbf09-3f87-4462-afdb-4b98ab34c72d","Type":"ContainerDied","Data":"8e083425913c42a2a4fa9e38ad5240377744f71fd8af8a3f50497ee845550da4"}
Sep 17 18:00:22.834238 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.834239    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"
Sep 17 18:00:22.834620 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.834297    2575 scope.go:117] "RemoveContainer" containerID="dddac0f5f372481943e081e4d885dc15921550eb6581affce5baba33d5000753"
Sep 17 18:00:22.842879 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.842859    2575 scope.go:117] "RemoveContainer" containerID="d3ee4355bbb7a794a15ca78034183a3c48c0696682acd500ee39a099d8e39e16"
Sep 17 18:00:22.850021 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.849998    2575 scope.go:117] "RemoveContainer" containerID="cdc10594ce8d7c1171fbff9207f4d8dd92b5d8054a2c1b75e7239a4ee8d87591"
Sep 17 18:00:22.856656 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.856630    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 18:00:22.857646 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.857615    2575 scope.go:117] "RemoveContainer" containerID="5e3a07aa181013b05f0eca9f1fa17714cdf3b4b04e1e8ef9deaa9b5ce17a9f91"
Sep 17 18:00:22.861503 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:22.861479    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b854445b6-qshtl"]
Sep 17 18:00:23.940081 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940047    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dx4r5/must-gather-mm4j8"]
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940318    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="storage-initializer"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="storage-initializer"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940341    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kserve-container"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940346    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kserve-container"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940352    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940358    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940364    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940369    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940377    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kube-rbac-proxy"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940383    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kube-rbac-proxy"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940390    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940395    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="agent"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940448    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kserve-container"
Sep 17 18:00:23.940449 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940455    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kube-rbac-proxy"
Sep 17 18:00:23.940902 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940461    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0da48c8-a108-49ed-9ede-8adfe3860bc8" containerName="kserve-container"
Sep 17 18:00:23.940902 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.940468    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" containerName="kube-rbac-proxy"
Sep 17 18:00:23.944897 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.944880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:23.947208 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.947183    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dx4r5\"/\"default-dockercfg-jd7vc\""
Sep 17 18:00:23.947353 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.947257    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dx4r5\"/\"openshift-service-ca.crt\""
Sep 17 18:00:23.947353 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.947324    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dx4r5\"/\"kube-root-ca.crt\""
Sep 17 18:00:23.952068 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:23.952036    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/must-gather-mm4j8"]
Sep 17 18:00:24.007150 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.007130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsrp\" (UniqueName: \"kubernetes.io/projected/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-kube-api-access-trsrp\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.007240 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.007168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-must-gather-output\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.107399 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.107367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-must-gather-output\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.107517 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.107437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsrp\" (UniqueName: \"kubernetes.io/projected/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-kube-api-access-trsrp\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.107719 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.107695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-must-gather-output\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.115658 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.115626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsrp\" (UniqueName: \"kubernetes.io/projected/3cf8b93a-7070-4f2a-b0b0-66a5398b010e-kube-api-access-trsrp\") pod \"must-gather-mm4j8\" (UID: \"3cf8b93a-7070-4f2a-b0b0-66a5398b010e\") " pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.254932 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.254873    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/must-gather-mm4j8"
Sep 17 18:00:24.524801 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.524774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/must-gather-mm4j8"]
Sep 17 18:00:24.527124 ip-10-0-134-52 kubenswrapper[2575]: W0917 18:00:24.527078    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cf8b93a_7070_4f2a_b0b0_66a5398b010e.slice/crio-fa4bf1ede05ceef9cd7926264711c38075e4646d528a2ad0bbdda891ff18844e WatchSource:0}: Error finding container fa4bf1ede05ceef9cd7926264711c38075e4646d528a2ad0bbdda891ff18844e: Status 404 returned error can't find the container with id fa4bf1ede05ceef9cd7926264711c38075e4646d528a2ad0bbdda891ff18844e
Sep 17 18:00:24.529126 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.529089    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:00:24.777063 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.776999    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0bbf09-3f87-4462-afdb-4b98ab34c72d" path="/var/lib/kubelet/pods/de0bbf09-3f87-4462-afdb-4b98ab34c72d/volumes"
Sep 17 18:00:24.843650 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:24.843622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/must-gather-mm4j8" event={"ID":"3cf8b93a-7070-4f2a-b0b0-66a5398b010e","Type":"ContainerStarted","Data":"fa4bf1ede05ceef9cd7926264711c38075e4646d528a2ad0bbdda891ff18844e"}
Sep 17 18:00:25.850771 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:25.850734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/must-gather-mm4j8" event={"ID":"3cf8b93a-7070-4f2a-b0b0-66a5398b010e","Type":"ContainerStarted","Data":"41f2b9026d63e3f461a517bb2472e8ec3ed6f7d2c2692fbc8206dfdff55fc707"}
Sep 17 18:00:25.851206 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:25.850828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/must-gather-mm4j8" event={"ID":"3cf8b93a-7070-4f2a-b0b0-66a5398b010e","Type":"ContainerStarted","Data":"2c744b5f33a0df224a25bdf834dab22745ca3b9d052499eb0134f9edc4e29c7e"}
Sep 17 18:00:25.867963 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:25.867922    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dx4r5/must-gather-mm4j8" podStartSLOduration=2.089585554 podStartE2EDuration="2.867908232s" totalImagesPullingTime="778.322678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:25.866821411 +0000 UTC m=+1597.233953308" watchObservedRunningTime="2026-09-17 18:00:25.867908232 +0000 UTC m=+1597.235040157"
Sep 17 18:00:26.777269 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:26.777234    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fxgnf_8caffdba-5c31-4028-8e2a-3574ca05bb23/global-pull-secret-syncer/0.log"
Sep 17 18:00:26.951461 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:26.951435    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f5d72_a2024083-28b8-4e85-8dab-49e3b84bc155/konnectivity-agent/0.log"
Sep 17 18:00:27.052786 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:27.052704    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-52.ec2.internal_afdef641cffa9a246bda35ec2b888902/haproxy/0.log"
Sep 17 18:00:31.390313 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.390256    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrtxz_319e8629-d769-4f2e-a130-da999bcf0a07/node-exporter/0.log"
Sep 17 18:00:31.422749 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.422724    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrtxz_319e8629-d769-4f2e-a130-da999bcf0a07/kube-rbac-proxy/0.log"
Sep 17 18:00:31.467198 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.467170    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrtxz_319e8629-d769-4f2e-a130-da999bcf0a07/init-textfile/0.log"
Sep 17 18:00:31.607628 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.607600    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-g9rkc_5f363bc4-50aa-4bb9-9ff4-3ce9426326bd/kube-rbac-proxy-main/0.log"
Sep 17 18:00:31.636962 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.636928    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-g9rkc_5f363bc4-50aa-4bb9-9ff4-3ce9426326bd/kube-rbac-proxy-self/0.log"
Sep 17 18:00:31.667777 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:31.667701    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-g9rkc_5f363bc4-50aa-4bb9-9ff4-3ce9426326bd/openshift-state-metrics/0.log"
Sep 17 18:00:34.499085 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.499054    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f44f854f-p2zzt_f919820f-f9bf-4d17-aef1-9b0f01290e3d/console/0.log"
Sep 17 18:00:34.576426 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.576388    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"]
Sep 17 18:00:34.581851 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.581805    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.588569 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.588540    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"]
Sep 17 18:00:34.693032 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.693002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-podres\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.693205 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.693042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-proc\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.693205 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.693059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-lib-modules\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.693205 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.693085    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzlr\" (UniqueName: \"kubernetes.io/projected/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-kube-api-access-nfzlr\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.693321 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.693225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-sys\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.793983 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.793900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-podres\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.793983 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.793948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-proc\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.793983 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.793972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-lib-modules\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-proc\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794077    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-lib-modules\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794079    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-podres\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzlr\" (UniqueName: \"kubernetes.io/projected/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-kube-api-access-nfzlr\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-sys\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.794298 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.794247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-sys\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.802818 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.802788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzlr\" (UniqueName: \"kubernetes.io/projected/f51edb1d-fddb-42cc-9b6c-eda0c19f26b8-kube-api-access-nfzlr\") pod \"perf-node-gather-daemonset-xgrz6\" (UID: \"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:34.895453 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:34.895423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:35.186393 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:35.186305    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"]
Sep 17 18:00:35.192208 ip-10-0-134-52 kubenswrapper[2575]: W0917 18:00:35.192178    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf51edb1d_fddb_42cc_9b6c_eda0c19f26b8.slice/crio-1f2e1ebc258396bac6b060b890fccd4e64867355a6ddbb4a40fec0d0f8b0cbfa WatchSource:0}: Error finding container 1f2e1ebc258396bac6b060b890fccd4e64867355a6ddbb4a40fec0d0f8b0cbfa: Status 404 returned error can't find the container with id 1f2e1ebc258396bac6b060b890fccd4e64867355a6ddbb4a40fec0d0f8b0cbfa
Sep 17 18:00:35.902400 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:35.902291    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6" event={"ID":"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8","Type":"ContainerStarted","Data":"9f4b22749cea5cbad4d59b2487810e043dc70c51c74654bbdadc68e4ab175a5e"}
Sep 17 18:00:35.902400 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:35.902361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6" event={"ID":"f51edb1d-fddb-42cc-9b6c-eda0c19f26b8","Type":"ContainerStarted","Data":"1f2e1ebc258396bac6b060b890fccd4e64867355a6ddbb4a40fec0d0f8b0cbfa"}
Sep 17 18:00:35.903002 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:35.902979    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:35.920718 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:35.920664    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6" podStartSLOduration=1.920644078 podStartE2EDuration="1.920644078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:35.91867783 +0000 UTC m=+1607.285809712" watchObservedRunningTime="2026-09-17 18:00:35.920644078 +0000 UTC m=+1607.287775960"
Sep 17 18:00:36.047007 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:36.046975    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88sj8_1773536c-31a2-436a-9f79-c5441b182f23/dns/0.log"
Sep 17 18:00:36.099911 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:36.099882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-88sj8_1773536c-31a2-436a-9f79-c5441b182f23/kube-rbac-proxy/0.log"
Sep 17 18:00:36.368816 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:36.368786    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xnbdh_56037b3a-b283-43e7-aad2-f0e19394874b/dns-node-resolver/0.log"
Sep 17 18:00:36.845305 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:36.845281    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c8n74_e0616c07-ca03-4208-a925-cba77fe73ec1/node-ca/0.log"
Sep 17 18:00:37.671812 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:37.671779    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66b4d4d798-wssc6_67a217b5-fc9b-4eb4-ac95-d68b96579ded/router/0.log"
Sep 17 18:00:38.077189 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:38.077116    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbmz9_282abe8e-7335-47ae-8ed9-ce6e2f1a36e5/serve-healthcheck-canary/0.log"
Sep 17 18:00:38.771531 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:38.771502    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6jkw_841de304-78b6-4f84-8b77-ff547166887d/kube-rbac-proxy/0.log"
Sep 17 18:00:38.801074 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:38.801042    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6jkw_841de304-78b6-4f84-8b77-ff547166887d/exporter/0.log"
Sep 17 18:00:38.827908 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:38.827882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6jkw_841de304-78b6-4f84-8b77-ff547166887d/extractor/0.log"
Sep 17 18:00:41.125230 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:41.125201    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-9b487cb79-cmxb5_4106a474-e143-40c1-bbfe-be7abf40ba54/manager/0.log"
Sep 17 18:00:41.168471 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:41.168439    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-kv6rc_98b7eb32-ce43-463b-8e6b-7f8f29718710/manager/0.log"
Sep 17 18:00:42.920967 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:42.920941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-xgrz6"
Sep 17 18:00:47.348750 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.348713    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-89kdm_e96db70e-2d64-41ad-902d-531efbc42295/kube-multus/0.log"
Sep 17 18:00:47.792979 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.792950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:00:47.826795 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.826769    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/egress-router-binary-copy/0.log"
Sep 17 18:00:47.855833 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.855812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/cni-plugins/0.log"
Sep 17 18:00:47.882758 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.882734    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/bond-cni-plugin/0.log"
Sep 17 18:00:47.907738 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.907722    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/routeoverride-cni/0.log"
Sep 17 18:00:47.934213 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.934174    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/whereabouts-cni-bincopy/0.log"
Sep 17 18:00:47.960035 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:47.960017    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgh7c_b8c30850-babf-4c71-8e02-ea922ec6732a/whereabouts-cni/0.log"
Sep 17 18:00:48.183316 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:48.183291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-twm78_2aff5a10-19db-45d6-8463-2fdfea258bbf/network-metrics-daemon/0.log"
Sep 17 18:00:48.217076 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:48.217059    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-twm78_2aff5a10-19db-45d6-8463-2fdfea258bbf/kube-rbac-proxy/0.log"
Sep 17 18:00:49.554119 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.554073    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-controller/0.log"
Sep 17 18:00:49.575414 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.575389    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/1.log"
Sep 17 18:00:49.575564 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.575430    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovn-acl-logging/0.log"
Sep 17 18:00:49.603278 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.603255    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/kube-rbac-proxy-node/0.log"
Sep 17 18:00:49.635118 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.635079    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:00:49.656351 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.656330    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/northd/0.log"
Sep 17 18:00:49.683334 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.683316    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/nbdb/0.log"
Sep 17 18:00:49.721102 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.721081    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/sbdb/0.log"
Sep 17 18:00:49.793170 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:49.793147    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbx6h_98409144-10a7-4d73-81f5-1575ffe6fc07/ovnkube-controller/0.log"
Sep 17 18:00:50.939629 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:50.939601    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vzdrl_053196ed-d593-4a70-a5cf-99164b6aca3d/network-check-target-container/0.log"
Sep 17 18:00:51.949279 ip-10-0-134-52 kubenswrapper[2575]: I0917 18:00:51.949185    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gz8xp_a00801ba-17a4-4fbc-aaa8-bbb5be9ed311/iptables-alerter/0.log"