Jul 10 18:58:33.586012 ip-10-0-139-20 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:58:33.981373 ip-10-0-139-20 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:33.981373 ip-10-0-139-20 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:58:33.981373 ip-10-0-139-20 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:33.981373 ip-10-0-139-20 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:58:33.981373 ip-10-0-139-20 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:33.983070 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.982984    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:58:33.987646 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987629    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:33.987646 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987646    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987650    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987653    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987657    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987660    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987664    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987666    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987671    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987675    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987678    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987680    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987683    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987686    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987688    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987691    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987694    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987697    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987700    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987708    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:33.987711 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987711    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987713    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987716    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987719    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987722    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987724    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987727    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987729    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987744    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987747    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987749    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987753    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987755    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987758    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987760    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987763    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987765    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987768    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987771    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:33.988176 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987773    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987776    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987778    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987780    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987783    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987785    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987788    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987790    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987792    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987795    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987797    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987799    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987802    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987804    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987807    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987810    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987813    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987815    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987817    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987820    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987822    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:33.988642 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987825    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987827    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987829    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987832    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987834    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987836    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987839    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987841    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987843    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987846    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987849    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987853    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987855    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987858    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987862    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987865    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987868    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987871    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987873    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:33.989205 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987876    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987878    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987881    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987883    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987885    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987888    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987890    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987894    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987898    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987901    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987904    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987906    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987909    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987911    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987914    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987916    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987919    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987921    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987923    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:33.989670 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.987926    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988618    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988626    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988630    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988635    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988637    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988641    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988644    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988647    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988650    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988652    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988655    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988657    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988660    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988662    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988665    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988667    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988670    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988672    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:33.990145 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988675    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988678    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988681    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988684    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988686    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988689    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988692    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988694    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988697    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988699    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988702    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988704    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988707    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988709    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988712    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988714    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988716    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988719    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988721    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988724    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:33.990606 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988726    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988729    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988745    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988748    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988750    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988753    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988755    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988757    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988760    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988762    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988764    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988767    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988770    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988773    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988775    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988778    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988780    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988783    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988785    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988787    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:33.991118 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988790    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988793    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988796    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988798    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988801    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988803    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988806    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988808    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988810    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988813    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988815    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988818    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988822    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988825    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988827    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988829    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988833    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988837    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988840    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:33.991607 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988842    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988845    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988847    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988850    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988852    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988855    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988858    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988861    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988863    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988866    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988868    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988872    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988875    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988878    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988881    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988884    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988887    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988889    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988892    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988894    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:33.992079 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988897    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.988900    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989265    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989270    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989274    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989276    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989284    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989287    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989290    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989292    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989295    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989297    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989300    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989302    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989305    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989308    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989310    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989316    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989319    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989321    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989324    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:33.992564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989326    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989329    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989333    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989336    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989339    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989341    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989344    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989346    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989349    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989351    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989353    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989356    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989358    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989361    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989363    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989365    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989368    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989370    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989373    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:33.993084 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989376    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989378    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989381    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989383    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989386    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989388    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989390    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989393    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989395    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989399    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989402    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989404    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989406    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989409    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989411    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989414    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989416    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989418    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989421    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:33.993592 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989424    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989426    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989429    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989431    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989433    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989436    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989438    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989440    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989443    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989445    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989448    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989450    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989455    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989459    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989461    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989464    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989467    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989470    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989472    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:33.994067 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989475    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989477    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989480    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989483    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989486    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989488    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989491    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989493    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989496    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989499    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989502    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989505    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989508    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989511    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989514    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989516    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989519    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989521    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989524    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:33.994657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989526    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989528    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989531    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.989533    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989597    2550 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989604    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989610    2550 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989614    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989619    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989622    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989626    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989629    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989632    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989635    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989638    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989641    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989646    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989649    2550 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989652    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989654    2550 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989657    2550 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989660    2550 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989664    2550 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:58:33.995142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989667    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989670    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989673    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989676    2550 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989680    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989682    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989686    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989688    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989691    2550 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989694    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989697    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989700    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989703    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989707    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989710    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989712    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989715    2550 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989717    2550 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989723    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989728    2550 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989744    2550 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989747    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989750    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989753    2550 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:58:33.995687 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989757    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989762    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989765    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989768    2550 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989770    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989773    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989776    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989779    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989781    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989784    2550 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989787    2550 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989790    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989793    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989795    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989798    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989801    2550 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989804    2550 flags.go:64] FLAG: --help="false"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989806    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-139-20.ec2.internal"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989809    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989812    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989815    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989818    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989824    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989826    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:58:33.996276 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989829    2550 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989832    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989834    2550 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989838    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989841    2550 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989843    2550 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989846    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989848    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989851    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989855    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989858    2550 flags.go:64] FLAG: --lock-file=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989861    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989863    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989866    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989871    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989874    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989876    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989879    2550 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989882    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989885    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989888    2550 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989890    2550 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989894    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989897    2550 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989901    2550 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:58:33.997077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989903    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989906    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989909    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989912    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989914    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989919    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989922    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989928    2550 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989931    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989934    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989937    2550 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989941    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989946    2550 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989949    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989952    2550 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989955    2550 flags.go:64] FLAG: --port="10250"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989957    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989961    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01de7cd2631921fc4"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989965    2550 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989967    2550 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989970    2550 flags.go:64] FLAG: --register-node="true"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989973    2550 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989976    2550 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989979    2550 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:58:33.998189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989982    2550 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989984    2550 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989993    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989996    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.989998    2550 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990001    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990004    2550 flags.go:64] FLAG: --runonce="false"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990006    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990009    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990012    2550 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990014    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990017    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990020    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990024    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990027    2550 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990029    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990032    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990034    2550 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990037    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990040    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990044    2550 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990047    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990051    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990054    2550 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990057    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:58:33.999223 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990061    2550 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990064    2550 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990067    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990069    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990072    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990075    2550 flags.go:64] FLAG: --v="2"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990079    2550 flags.go:64] FLAG: --version="false"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990082    2550 flags.go:64] FLAG: --vmodule=""
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990086    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.990089    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990197    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990201    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990204    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990207    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990209    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990212    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990215    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990218    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990220    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990223    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990227    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:34.000313 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990230    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990232    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990235    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990237    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990240    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990242    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990245    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990247    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990250    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990252    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990255    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990258    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990261    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990263    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990265    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990268    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990270    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990273    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990275    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990277    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:34.001189 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990280    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990282    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990286    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990289    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990292    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990309    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990312    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990315    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990318    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990321    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990323    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990327    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990330    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990332    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990335    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990338    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990340    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990344    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990347    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:34.001674 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990350    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990353    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990356    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990359    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990362    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990364    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990367    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990370    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990373    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990376    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990378    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990381    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990383    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990386    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990388    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990391    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990393    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990396    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990398    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990400    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:34.002433 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990403    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990405    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990408    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990410    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990416    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990419    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990421    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990424    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990426    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990429    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990432    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990434    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990437    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990439    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990442    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990444    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990447    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990449    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990451    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990454    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:34.003138 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990456    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990459    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990461    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990463    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990466    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990470    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990473    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990475    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.990478    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.991163    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.999478    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.999494    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.999520    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:33.999528    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:58:34.003843 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999622    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999629    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999635    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999640    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999644    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999649    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999663    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999671    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999676    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999680    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999684    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999689    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999701    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999707    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999712    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999716    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999721    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999725    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999729    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:34.004338 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999752    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999756    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999760    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999764    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999768    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999772    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999776    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999780    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999784    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999788    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999793    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999797    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999801    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999805    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999809    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999813    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999817    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999821    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999825    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999829    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:34.005022 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999833    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999837    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999842    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999846    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999850    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999854    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999866    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999871    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999875    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999879    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999883    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999887    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999891    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999895    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999899    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999903    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999909    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999915    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999919    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999924    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:34.005657 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999928    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999932    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999937    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999942    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999947    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999951    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999955    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999959    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999963    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999967    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999971    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999975    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999979    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999983    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999987    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999991    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:33.999995    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000002    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000008    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:34.006211 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000022    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000027    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000031    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000035    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000039    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000043    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000047    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000051    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000055    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000059    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000064    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000068    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000072    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000076    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000080    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000085    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000089    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000093    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000097    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000101    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:34.006684 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000105    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.000113    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000332    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000341    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000346    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000350    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000354    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000359    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000363    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000368    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000371    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000375    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000380    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000392    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000404    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:34.007227 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000408    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000412    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000416    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000420    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000425    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000429    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000433    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000437    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000441    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000445    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000449    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000453    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000457    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000462    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000466    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000470    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000474    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000478    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000482    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000487    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:34.007596 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000490    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000495    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000499    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000503    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000507    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000511    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000515    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000519    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000522    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000528    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000533    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000545    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000549    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000554    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000558    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000562    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000565    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000570    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000574    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000579    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:34.008109 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000582    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000586    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000590    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000594    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000599    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000603    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000608    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000612    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000616    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000620    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000624    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000628    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000632    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000636    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000640    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000644    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000648    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000652    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000656    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000660    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:34.008614 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000664    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000669    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000672    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000677    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000688    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000692    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000696    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000700    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000704    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000708    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000712    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000716    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000720    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000723    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000730    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000755    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000760    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000766    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000770    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:34.009219 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000775    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000780    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000784    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000788    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000794    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000800    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:34.000804    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.000812    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.000964    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.006648    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.008059    2550 server.go:1019] "Starting client certificate rotation"
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.008152    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:34.009692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.008189    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:34.010694 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.010680    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:34.011786 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.011767    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:34.026217 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.026196    2550 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:58:34.028706 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.028692    2550 log.go:25] "Validated CRI v1 image API"
Jul 10 18:58:34.029682 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.029669    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:58:34.034812 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.034790    2550 fs.go:135] Filesystem UUIDs: map[037f812a-0d6c-48c2-a337-5ae2a40f20bb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fefccd7e-83cd-4ea6-a433-9fb9466be99b:/dev/nvme0n1p4]
Jul 10 18:58:34.034899 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.034811    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:58:34.040646 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.040534    2550 manager.go:216] Machine: {Timestamp:2026-07-10 18:58:34.039112727 +0000 UTC m=+0.337464086 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101459 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d43d2d2ae9f70decca3922222e622 SystemUUID:ec2d43d2-d2ae-9f70-decc-a3922222e622 BootID:f2fd5569-6d87-4c83-b4b3-ab0d6fa830e7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:da:1d:2a:c7:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:da:1d:2a:c7:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:d0:3b:71:8b:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:58:34.040646 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.040638    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:58:34.040834 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.040762    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:58:34.041967 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.041949    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:34.043776 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.043752    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:58:34.043917 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.043780    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-20.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:58:34.044595 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.044583    2550 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:58:34.044626 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.044599    2550 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:58:34.044626 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.044606    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:34.045315 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.045305    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:34.047215 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.047205    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:58:34.052441 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052428    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:34.052590 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052580    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:58:34.052620 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052613    2550 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:58:34.052646 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052630    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:58:34.052690 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052650    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:58:34.052690 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052661    2550 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:58:34.052690 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.052671    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:58:34.053059 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.053045    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:34.053095 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.053068    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:34.056072 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.055862    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:58:34.058393 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058377    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:58:34.058472 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058416    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:58:34.058609 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058596    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:58:34.058673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058623    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:58:34.058673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058634    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:58:34.058673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058643    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:58:34.058673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058655    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:58:34.058673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.058664    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:58:34.059232 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059221    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:58:34.059292 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059235    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:58:34.059292 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059246    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:58:34.059292 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059255    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:58:34.059292 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059268    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:58:34.059292 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.059280    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:58:34.060019 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.060008    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:58:34.060067 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.060021    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:58:34.063371 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.063351    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-20.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:58:34.063371 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.063364    2550 server.go:1268] "Started kubelet"
Jul 10 18:58:34.063513 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.063431    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:58:34.063513 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.063438    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:58:34.063513 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.063457    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:58:34.063513 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.063479    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:58:34.063691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.063527    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:58:34.064253 ip-10-0-139-20 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:58:34.065089 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.065050    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:58:34.065763 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.065729    2550 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:58:34.067841 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.067820    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm85f"
Jul 10 18:58:34.070390 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.070372    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:58:34.070515 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.069758    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-20.ec2.internal.18c102b7621683b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-20.ec2.internal,UID:ip-10-0-139-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-20.ec2.internal,},FirstTimestamp:2026-07-10 18:58:34.063340467 +0000 UTC m=+0.361691828,LastTimestamp:2026-07-10 18:58:34.063340467 +0000 UTC m=+0.361691828,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-20.ec2.internal,}"
Jul 10 18:58:34.072464 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.072443    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:58:34.072464 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.072449    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:34.072621 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.072539    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.073224 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073208    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:58:34.073224 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073208    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:58:34.073366 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073232    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:58:34.073366 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073338    2550 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:58:34.073912 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073894    2550 factory.go:55] Registering systemd factory
Jul 10 18:58:34.073912 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073913    2550 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:58:34.074036 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.073985    2550 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:58:34.074175 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074163    2550 factory.go:153] Registering CRI-O factory
Jul 10 18:58:34.074230 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074180    2550 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:58:34.074274 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074264    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:58:34.074315 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074286    2550 factory.go:103] Registering Raw factory
Jul 10 18:58:34.074315 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074297    2550 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:58:34.074666 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.074654    2550 manager.go:349] Starting recovery of all containers
Jul 10 18:58:34.075996 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.075969    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 18:58:34.076104 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.076082    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 18:58:34.076320 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.076303    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm85f"
Jul 10 18:58:34.086265 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.086244    2550 manager.go:354] Recovery completed
Jul 10 18:58:34.087550 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.087532    2550 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 10 18:58:34.090311 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.090298    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.093960 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.093945    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.094035 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.093971    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.094035 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.093982    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.094518 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.094504    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:58:34.094518 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.094512    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.094518 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.094518    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:58:34.094688 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.094538    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:34.095241 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095227    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.095336 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095253    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.095336 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095267    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.095666 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095654    2550 policy_none.go:49] "None policy: Start"
Jul 10 18:58:34.095723 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095673    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:58:34.095723 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.095687    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:58:34.096419 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.096343    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-20.ec2.internal.18c102b763e9b841  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-20.ec2.internal,UID:ip-10-0-139-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-20.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-20.ec2.internal,},FirstTimestamp:2026-07-10 18:58:34.093959233 +0000 UTC m=+0.392310593,LastTimestamp:2026-07-10 18:58:34.093959233 +0000 UTC m=+0.392310593,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-20.ec2.internal,}"
Jul 10 18:58:34.096851 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.096839    2550 policy_none.go:47] "Start"
Jul 10 18:58:34.133295 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133279    2550 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:58:34.133381 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.133320    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:58:34.133381 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133330    2550 server.go:87] "Starting device plugin registration server"
Jul 10 18:58:34.133601 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133588    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:58:34.133679 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133604    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:58:34.133749 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133684    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:58:34.133824 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133782    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:58:34.133824 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.133791    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:58:34.134338 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.134308    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:58:34.134429 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.134360    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.234253 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.234184    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.235721 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.235393    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.235721 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.235425    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.235721 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.235438    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.235721 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.235465    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.245469 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.245454    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.245537 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.245476    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-20.ec2.internal\": node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.267750 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.267692    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:58:34.269068 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.269048    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:58:34.269153 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.269083    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:58:34.269153 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.269101    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:58:34.269153 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.269139    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:58:34.273374 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.273355    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.276211 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.276197    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:34.369749 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.369683    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"]
Jul 10 18:58:34.369869 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.369795    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.370792 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.370779    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.370862 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.370810    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.370862 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.370826    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.372879 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.372865    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.373005 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.372993    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.373047 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373021    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.373822 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.373789    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.373933 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373880    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.373933 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373908    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.373933 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373926    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.374069 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373953    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.374069 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373969    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.374069 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.373982    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.375448 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.375428    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.375520 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.375464    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.375520 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.375491    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d70378dbc438c3228b7eaf66f8d5709-config\") pod \"kube-apiserver-proxy-ip-10-0-139-20.ec2.internal\" (UID: \"1d70378dbc438c3228b7eaf66f8d5709\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.376116 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.376098    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.376194 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.376130    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:34.376877 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.376858    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:34.376971 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.376886    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:34.376971 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.376895    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:34.409341 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.409322    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-20.ec2.internal\" not found" node="ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.413871 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.413855    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-20.ec2.internal\" not found" node="ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.473976 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.473953    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.476143 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.476196 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476151    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.476196 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476167    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d70378dbc438c3228b7eaf66f8d5709-config\") pod \"kube-apiserver-proxy-ip-10-0-139-20.ec2.internal\" (UID: \"1d70378dbc438c3228b7eaf66f8d5709\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.476263 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476194    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d70378dbc438c3228b7eaf66f8d5709-config\") pod \"kube-apiserver-proxy-ip-10-0-139-20.ec2.internal\" (UID: \"1d70378dbc438c3228b7eaf66f8d5709\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.476263 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476214    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.476263 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.476227    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc1b2a120fe281acd7e52132f4adecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal\" (UID: \"cfc1b2a120fe281acd7e52132f4adecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.574531 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.574460    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.674917 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.674887    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.712205 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.712186    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.717651 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.717633    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:34.775441 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.775411    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.876064 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.875993    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:34.918443 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:34.918410    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:34.977090 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:34.977067    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:35.007289 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.007273    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:58:35.007852 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.007409    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:35.007852 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.007449    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:35.077286 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:35.077254    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:35.081155 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.081123    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:53:34 +0000 UTC" deadline="2028-02-18 04:31:11.468239455 +0000 UTC"
Jul 10 18:58:35.081155 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.081154    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14097h32m36.387088866s"
Jul 10 18:58:35.178406 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:35.178325    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-20.ec2.internal\" not found"
Jul 10 18:58:35.221893 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.221861    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:35.228968 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:35.228938    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d70378dbc438c3228b7eaf66f8d5709.slice/crio-491b2a8a27af6077e9888641a2448898c74d55b6aaaf03220fc031f6d0e1b9f3 WatchSource:0}: Error finding container 491b2a8a27af6077e9888641a2448898c74d55b6aaaf03220fc031f6d0e1b9f3: Status 404 returned error can't find the container with id 491b2a8a27af6077e9888641a2448898c74d55b6aaaf03220fc031f6d0e1b9f3
Jul 10 18:58:35.229217 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:35.229203    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc1b2a120fe281acd7e52132f4adecf.slice/crio-4982dbb98c9346cdee94ae78b6a8bd143cb994c6d55d86972e954969ba6c4214 WatchSource:0}: Error finding container 4982dbb98c9346cdee94ae78b6a8bd143cb994c6d55d86972e954969ba6c4214: Status 404 returned error can't find the container with id 4982dbb98c9346cdee94ae78b6a8bd143cb994c6d55d86972e954969ba6c4214
Jul 10 18:58:35.233914 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.233897    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:58:35.272570 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.272547    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:35.272716 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.272615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal" event={"ID":"1d70378dbc438c3228b7eaf66f8d5709","Type":"ContainerStarted","Data":"491b2a8a27af6077e9888641a2448898c74d55b6aaaf03220fc031f6d0e1b9f3"}
Jul 10 18:58:35.273814 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.273791    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal" event={"ID":"cfc1b2a120fe281acd7e52132f4adecf","Type":"ContainerStarted","Data":"4982dbb98c9346cdee94ae78b6a8bd143cb994c6d55d86972e954969ba6c4214"}
Jul 10 18:58:35.288176 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.288157    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:35.288988 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.288973    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal"
Jul 10 18:58:35.298799 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.298784    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:35.306695 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.306677    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:35.981206 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:35.981177    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:36.055591 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.055557    2550 apiserver.go:52] "Watching apiserver"
Jul 10 18:58:36.064704 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.064669    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:58:36.065110 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.065081    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-sfhhk","openshift-network-operator/iptables-alerter-k9755","kube-system/global-pull-secret-syncer-w9gm9","kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j","openshift-multus/multus-4qbhw","openshift-multus/multus-additional-cni-plugins-pzkb7","openshift-multus/network-metrics-daemon-5mrgz","openshift-ovn-kubernetes/ovnkube-node-9x886","kube-system/konnectivity-agent-tz7r4","openshift-cluster-node-tuning-operator/tuned-pfg5n","openshift-dns/node-resolver-jmnrl","openshift-image-registry/node-ca-zsw4k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal"]
Jul 10 18:58:36.066829 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.066805    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.068577 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.068326    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.069747 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.069723    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:58:36.069996 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.069973    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4n8ln\""
Jul 10 18:58:36.069996 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.069986    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:58:36.070555 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.070533    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.070660 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.070624    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:36.071993 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.071970    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.072084 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.072042    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.072989    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.073002    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.073026    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nchj8\""
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.073092    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.073649    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.073800 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.073760    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:36.075243 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.075222    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.076774 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.076751    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.078278 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078256    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:58:36.078373 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078344    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.078485 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078462    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ld2n4\""
Jul 10 18:58:36.078692 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078669    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:58:36.078798 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078698    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.078872 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078827    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.078872 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078860    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.078998 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.078983    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.079390 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.079362    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fdc8w\""
Jul 10 18:58:36.079488 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.079403    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:58:36.079488 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.079456    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:58:36.079611 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.079515    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:58:36.079693 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.079675    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:58:36.080614 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.080595    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cx7z\""
Jul 10 18:58:36.080821 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.080799    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.081026 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.081012    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:58:36.081189 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.081173    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.081769 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.081736    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.082149 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.082128    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jtvmh\""
Jul 10 18:58:36.082540 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.082486    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:36.082634 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.082564    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:36.082742 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.082723    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:58:36.082811 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.082743    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:58:36.083779 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083739    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d42dd62c-6e03-4e8d-bc39-3021e33abd05-agent-certs\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.083779 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.083935 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083844    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr9j\" (UniqueName: \"kubernetes.io/projected/bad76711-2921-4515-b2f4-26554d0783aa-kube-api-access-nsr9j\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.083935 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-device-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.084043 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083922    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.084043 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b103d35-f097-45f6-906b-11016846795f-host\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.084043 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.083989    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-kubelet-config\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.084043 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.084036    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-registration-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.084256 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.084071    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjl5\" (UniqueName: \"kubernetes.io/projected/9b103d35-f097-45f6-906b-11016846795f-kube-api-access-rgjl5\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.084256 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.084095    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-dbus\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.086268 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.086239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.086437 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.086413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.086537 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.086485    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.086537 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.086494    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-socket-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.087916 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.087893    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.088413 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.088386    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.088566 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.088548    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:58:36.091081 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.091055    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-clx9t\""
Jul 10 18:58:36.091445 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.091422    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093399    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-sys-fs\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df47n\" (UniqueName: \"kubernetes.io/projected/83d48c51-3584-4f34-bd45-8235ae7c2e38-kube-api-access-df47n\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093525    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d42dd62c-6e03-4e8d-bc39-3021e33abd05-konnectivity-ca\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093557    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b103d35-f097-45f6-906b-11016846795f-serviceca\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093970    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.093994    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pv2jg\""
Jul 10 18:58:36.094482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.094226    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.095370 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.094754    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8ckt8\""
Jul 10 18:58:36.095370 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.095016    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:58:36.095370 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.095053    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:36.095932 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.095908    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:58:36.112750 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.112642    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdtq7"
Jul 10 18:58:36.122396 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.122091    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdtq7"
Jul 10 18:58:36.174550 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.174528    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:58:36.194171 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194142    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d42dd62c-6e03-4e8d-bc39-3021e33abd05-agent-certs\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.194322 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194183    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-device-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.194322 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194222    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.194322 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194251    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-tmp\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.194322 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194282    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b103d35-f097-45f6-906b-11016846795f-host\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.194322 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194284    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-device-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-node-log\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194353    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194380    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovn-node-metrics-cert\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194385    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b103d35-f097-45f6-906b-11016846795f-host\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb296e4e-b237-491f-9d54-f4363d7cfdcb-iptables-alerter-script\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194452    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwlz\" (UniqueName: \"kubernetes.io/projected/be349af5-8724-4dd3-94ed-7ac76841433e-kube-api-access-bvwlz\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194483    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-netns\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-hostroot\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194532    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-multus-certs\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.194592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194561    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d42dd62c-6e03-4e8d-bc39-3021e33abd05-konnectivity-ca\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194588    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-log-socket\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194608    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194634    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-config\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194658    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb296e4e-b237-491f-9d54-f4363d7cfdcb-host-slash\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194679    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-hosts-file\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194701    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-os-release\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-etc-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194740    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-systemd\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-lib-modules\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194787    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-etc-kubernetes\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194832    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-registration-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194878    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-var-lib-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194902    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-env-overrides\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194927    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgc75\" (UniqueName: \"kubernetes.io/projected/56678559-0a6b-42b5-b192-5d8bcd4a36ed-kube-api-access-lgc75\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194971    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-system-cni-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.194996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-system-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.195087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195020    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-cni-binary-copy\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195024    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-registration-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195051    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195083    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df47n\" (UniqueName: \"kubernetes.io/projected/83d48c51-3584-4f34-bd45-8235ae7c2e38-kube-api-access-df47n\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195108    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cnibin\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195180    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-binary-copy\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d42dd62c-6e03-4e8d-bc39-3021e33abd05-konnectivity-ca\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195264    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-conf\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195273    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-multus\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195334    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-kubelet\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-os-release\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195397    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fklw\" (UniqueName: \"kubernetes.io/projected/bb296e4e-b237-491f-9d54-f4363d7cfdcb-kube-api-access-7fklw\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195427    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-run\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195469    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-etc-tuned\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195531    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-tmp-dir\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.196262 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195567    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195596    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-modprobe-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195625    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195651    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-sys\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.195674    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195676    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-var-lib-kubelet\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.195786    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:36.69572801 +0000 UTC m=+2.994079377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-cnibin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-kubelet\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195877    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-kubelet-config\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195906    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-bin\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195935    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-dbus\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195962    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.195961    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-kubelet-config\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196011    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-socket-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196043    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-sys-fs\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.196055    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:36.197267 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196088    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/537fede7-1201-4688-9690-276e542b8f13-dbus\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196071    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtq56\" (UniqueName: \"kubernetes.io/projected/5691beb4-1cf8-4df5-b6a4-f11bb372c145-kube-api-access-jtq56\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196127    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-k8s-cni-cncf-io\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.196149    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:58:36.696121346 +0000 UTC m=+2.994472698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196179    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcx6w\" (UniqueName: \"kubernetes.io/projected/39adceed-d61c-41d6-abf8-2c341e30d3f9-kube-api-access-jcx6w\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196207    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-socket-dir\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196218    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-sys-fs\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196207    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196256    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196285    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-host\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196311    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-socket-dir-parent\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196336    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-bin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196365    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr9j\" (UniqueName: \"kubernetes.io/projected/bad76711-2921-4515-b2f4-26554d0783aa-kube-api-access-nsr9j\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196393    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196418    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-conf-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196443    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-daemon-config\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.198181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196467    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-slash\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196492    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83d48c51-3584-4f34-bd45-8235ae7c2e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196520    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-kubernetes\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196624    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196653    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196682    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysconfig\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjl5\" (UniqueName: \"kubernetes.io/projected/9b103d35-f097-45f6-906b-11016846795f-kube-api-access-rgjl5\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196735    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-systemd-units\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196760    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-netns\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196784    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-systemd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196831    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b103d35-f097-45f6-906b-11016846795f-serviceca\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-ovn\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196898    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-netd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-script-lib\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.196981    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnxq\" (UniqueName: \"kubernetes.io/projected/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-kube-api-access-gqnxq\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.199136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.197502    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b103d35-f097-45f6-906b-11016846795f-serviceca\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.199872 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.199723    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d42dd62c-6e03-4e8d-bc39-3021e33abd05-agent-certs\") pod \"konnectivity-agent-tz7r4\" (UID: \"d42dd62c-6e03-4e8d-bc39-3021e33abd05\") " pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.210982 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.210956    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr9j\" (UniqueName: \"kubernetes.io/projected/bad76711-2921-4515-b2f4-26554d0783aa-kube-api-access-nsr9j\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.211579 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.211534    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df47n\" (UniqueName: \"kubernetes.io/projected/83d48c51-3584-4f34-bd45-8235ae7c2e38-kube-api-access-df47n\") pod \"aws-ebs-csi-driver-node-wlh6j\" (UID: \"83d48c51-3584-4f34-bd45-8235ae7c2e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.216149 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.216121    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjl5\" (UniqueName: \"kubernetes.io/projected/9b103d35-f097-45f6-906b-11016846795f-kube-api-access-rgjl5\") pod \"node-ca-zsw4k\" (UID: \"9b103d35-f097-45f6-906b-11016846795f\") " pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298099    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-system-cni-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298146    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-system-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-cni-binary-copy\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cnibin\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298226    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-system-cni-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298248    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-system-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298265    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-binary-copy\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298293    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-conf\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298307    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cnibin\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298318    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-multus\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298346    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-kubelet\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298367    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-os-release\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298384    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fklw\" (UniqueName: \"kubernetes.io/projected/bb296e4e-b237-491f-9d54-f4363d7cfdcb-kube-api-access-7fklw\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298402    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-run\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-etc-tuned\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298447    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-tmp-dir\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.298804 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-modprobe-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298499    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298515    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-sys\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298531    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-var-lib-kubelet\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298557    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-cnibin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298577    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-kubelet\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298594    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-bin\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298622    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtq56\" (UniqueName: \"kubernetes.io/projected/5691beb4-1cf8-4df5-b6a4-f11bb372c145-kube-api-access-jtq56\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298638    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-k8s-cni-cncf-io\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298655    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcx6w\" (UniqueName: \"kubernetes.io/projected/39adceed-d61c-41d6-abf8-2c341e30d3f9-kube-api-access-jcx6w\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298679    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298731    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-host\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298758    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-socket-dir-parent\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298779    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-bin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-conf-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298850    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-daemon-config\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298855    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-cni-binary-copy\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.299793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298871    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-slash\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298894    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298925    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-binary-copy\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-kubernetes\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298959    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-multus\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298967    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-conf\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298969    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299001    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-kubelet\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299020    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299033    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-host\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299040    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-socket-dir-parent\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299048    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysconfig\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299064    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-os-release\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299080    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-systemd-units\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299095    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-cni-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299095    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-cni-bin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-netns\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299134    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-systemd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299134    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-conf-dir\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299141    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-netns\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299153    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-ovn\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299181    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-netd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299199    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-script-lib\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnxq\" (UniqueName: \"kubernetes.io/projected/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-kube-api-access-gqnxq\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299234    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299244    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-var-lib-kubelet\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299251    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-tmp\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299299    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-node-log\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299334    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299356    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovn-node-metrics-cert\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299374    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb296e4e-b237-491f-9d54-f4363d7cfdcb-iptables-alerter-script\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299392    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwlz\" (UniqueName: \"kubernetes.io/projected/be349af5-8724-4dd3-94ed-7ac76841433e-kube-api-access-bvwlz\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299410    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-netns\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.301859 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299427    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-hostroot\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-multus-certs\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-log-socket\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299479    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-config\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299480    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-tmp-dir\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299497    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb296e4e-b237-491f-9d54-f4363d7cfdcb-host-slash\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-hosts-file\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-os-release\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299568    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-etc-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299585    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-systemd\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299601    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-lib-modules\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299619    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-etc-kubernetes\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299637    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-var-lib-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-env-overrides\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299655    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39adceed-d61c-41d6-abf8-2c341e30d3f9-multus-daemon-config\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299673    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgc75\" (UniqueName: \"kubernetes.io/projected/56678559-0a6b-42b5-b192-5d8bcd4a36ed-kube-api-access-lgc75\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299725    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-slash\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299902    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5691beb4-1cf8-4df5-b6a4-f11bb372c145-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.302504 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.299999    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-k8s-cni-cncf-io\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-cnibin\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300366    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-var-lib-kubelet\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300418    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-bin\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300433    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300435    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.298922    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-run\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300475    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-sys\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300494    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-kubernetes\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300518    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-ovn\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-node-log\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300561    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb296e4e-b237-491f-9d54-f4363d7cfdcb-iptables-alerter-script\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300597    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300655    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-host-cni-netd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300742    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysctl-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300842    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-sysconfig\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300907    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-systemd-units\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.303459 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-modprobe-d\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300967    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-hosts-file\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301020    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-multus-certs\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301066    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-host-run-netns\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.300521    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-run-systemd\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301092    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-hostroot\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301199    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5691beb4-1cf8-4df5-b6a4-f11bb372c145-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301206    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb296e4e-b237-491f-9d54-f4363d7cfdcb-host-slash\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301269    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-var-lib-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301292    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-os-release\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301298    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-lib-modules\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301313    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-etc-openvswitch\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301343    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/56678559-0a6b-42b5-b192-5d8bcd4a36ed-log-socket\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301358    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39adceed-d61c-41d6-abf8-2c341e30d3f9-etc-kubernetes\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301359    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-script-lib\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301360    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be349af5-8724-4dd3-94ed-7ac76841433e-etc-systemd\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301528    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovnkube-config\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.301728    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/56678559-0a6b-42b5-b192-5d8bcd4a36ed-env-overrides\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.304691 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.302541    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-etc-tuned\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.305884 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.303096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/56678559-0a6b-42b5-b192-5d8bcd4a36ed-ovn-node-metrics-cert\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.305884 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.303339    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be349af5-8724-4dd3-94ed-7ac76841433e-tmp\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.308080 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.308053    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:36.308191 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.308085    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:36.308191 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.308099    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:36.308191 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.308174    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:36.808152032 +0000 UTC m=+3.106503381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:36.311431 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.311396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fklw\" (UniqueName: \"kubernetes.io/projected/bb296e4e-b237-491f-9d54-f4363d7cfdcb-kube-api-access-7fklw\") pod \"iptables-alerter-k9755\" (UID: \"bb296e4e-b237-491f-9d54-f4363d7cfdcb\") " pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.312588 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.312564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgc75\" (UniqueName: \"kubernetes.io/projected/56678559-0a6b-42b5-b192-5d8bcd4a36ed-kube-api-access-lgc75\") pod \"ovnkube-node-9x886\" (UID: \"56678559-0a6b-42b5-b192-5d8bcd4a36ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.312993 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.312956    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcx6w\" (UniqueName: \"kubernetes.io/projected/39adceed-d61c-41d6-abf8-2c341e30d3f9-kube-api-access-jcx6w\") pod \"multus-4qbhw\" (UID: \"39adceed-d61c-41d6-abf8-2c341e30d3f9\") " pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.315348 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.315327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtq56\" (UniqueName: \"kubernetes.io/projected/5691beb4-1cf8-4df5-b6a4-f11bb372c145-kube-api-access-jtq56\") pod \"multus-additional-cni-plugins-pzkb7\" (UID: \"5691beb4-1cf8-4df5-b6a4-f11bb372c145\") " pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.320683 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.320654    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwlz\" (UniqueName: \"kubernetes.io/projected/be349af5-8724-4dd3-94ed-7ac76841433e-kube-api-access-bvwlz\") pod \"tuned-pfg5n\" (UID: \"be349af5-8724-4dd3-94ed-7ac76841433e\") " pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.320823 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.320709    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnxq\" (UniqueName: \"kubernetes.io/projected/a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422-kube-api-access-gqnxq\") pod \"node-resolver-jmnrl\" (UID: \"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422\") " pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.382440 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.382413    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:36.393290 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.393266    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsw4k"
Jul 10 18:58:36.411324 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.411288    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j"
Jul 10 18:58:36.416246 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.416217    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4qbhw"
Jul 10 18:58:36.424156 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.424131    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:58:36.433173 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.433156    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pzkb7"
Jul 10 18:58:36.441011 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.440986    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9755"
Jul 10 18:58:36.450033 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.450012    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pfg5n"
Jul 10 18:58:36.456856 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.456836    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jmnrl"
Jul 10 18:58:36.702515 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.702426    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:36.702515 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.702495    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:36.702712 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.702564    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:36.702712 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.702622    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:36.702712 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.702655    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:37.70263642 +0000 UTC m=+4.000987774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:36.702712 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.702680    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:58:37.702669282 +0000 UTC m=+4.001020636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:36.904321 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:36.904285    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:36.904517 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.904492    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:36.904590 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.904529    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:36.904590 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.904544    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:36.904692 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:36.904618    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:37.904596928 +0000 UTC m=+4.202948300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:37.127059 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.126950    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:36 +0000 UTC" deadline="2027-12-05 20:13:41.113354267 +0000 UTC"
Jul 10 18:58:37.127059 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.127006    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12313h15m3.986354103s"
Jul 10 18:58:37.274701 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.270030    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:37.274701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.270239    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:37.288943 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.288917    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b2a3f9_d5b8_4242_a5e3_b1b5fd6dc422.slice/crio-bebd8dd75abd40f0ad3c4fe71482210ac0430c44104ed632e891c89fc4d0ac99 WatchSource:0}: Error finding container bebd8dd75abd40f0ad3c4fe71482210ac0430c44104ed632e891c89fc4d0ac99: Status 404 returned error can't find the container with id bebd8dd75abd40f0ad3c4fe71482210ac0430c44104ed632e891c89fc4d0ac99
Jul 10 18:58:37.293979 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.293850    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb296e4e_b237_491f_9d54_f4363d7cfdcb.slice/crio-0c410ab53a705efb0fc70355d369f6b202b972d1292dc8c638e1cfd0bac5e0a9 WatchSource:0}: Error finding container 0c410ab53a705efb0fc70355d369f6b202b972d1292dc8c638e1cfd0bac5e0a9: Status 404 returned error can't find the container with id 0c410ab53a705efb0fc70355d369f6b202b972d1292dc8c638e1cfd0bac5e0a9
Jul 10 18:58:37.297783 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.297442    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5691beb4_1cf8_4df5_b6a4_f11bb372c145.slice/crio-18e6703f6e685a60572f7a69ca81c9889c3d9470583a8dd6a86e1af8bacf43d8 WatchSource:0}: Error finding container 18e6703f6e685a60572f7a69ca81c9889c3d9470583a8dd6a86e1af8bacf43d8: Status 404 returned error can't find the container with id 18e6703f6e685a60572f7a69ca81c9889c3d9470583a8dd6a86e1af8bacf43d8
Jul 10 18:58:37.301115 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.300566    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe349af5_8724_4dd3_94ed_7ac76841433e.slice/crio-bb9a08c5990ecc0089026eaa3f25a3766933632b9e97ea3715c674cf77534380 WatchSource:0}: Error finding container bb9a08c5990ecc0089026eaa3f25a3766933632b9e97ea3715c674cf77534380: Status 404 returned error can't find the container with id bb9a08c5990ecc0089026eaa3f25a3766933632b9e97ea3715c674cf77534380
Jul 10 18:58:37.301855 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.301833    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d48c51_3584_4f34_bd45_8235ae7c2e38.slice/crio-ebb4c84bb802e2c1d554516d7f6b4059fa8fe0d904e4e5fe0db90bfa7692ef15 WatchSource:0}: Error finding container ebb4c84bb802e2c1d554516d7f6b4059fa8fe0d904e4e5fe0db90bfa7692ef15: Status 404 returned error can't find the container with id ebb4c84bb802e2c1d554516d7f6b4059fa8fe0d904e4e5fe0db90bfa7692ef15
Jul 10 18:58:37.303399 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.303085    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42dd62c_6e03_4e8d_bc39_3021e33abd05.slice/crio-43549396d242729151064efd272f459afa70995ae3497d72513d3052d8fa043d WatchSource:0}: Error finding container 43549396d242729151064efd272f459afa70995ae3497d72513d3052d8fa043d: Status 404 returned error can't find the container with id 43549396d242729151064efd272f459afa70995ae3497d72513d3052d8fa043d
Jul 10 18:58:37.303597 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.303536    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b103d35_f097_45f6_906b_11016846795f.slice/crio-7d309d384ad3b2cf8016782c419d54abb9eda888ce7743c11b4c5ef66f6c1976 WatchSource:0}: Error finding container 7d309d384ad3b2cf8016782c419d54abb9eda888ce7743c11b4c5ef66f6c1976: Status 404 returned error can't find the container with id 7d309d384ad3b2cf8016782c419d54abb9eda888ce7743c11b4c5ef66f6c1976
Jul 10 18:58:37.306132 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:58:37.305733    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56678559_0a6b_42b5_b192_5d8bcd4a36ed.slice/crio-d76d54411a62a25524b83578e12dd9a8bf002c4b19eae6348909b61befdd30a4 WatchSource:0}: Error finding container d76d54411a62a25524b83578e12dd9a8bf002c4b19eae6348909b61befdd30a4: Status 404 returned error can't find the container with id d76d54411a62a25524b83578e12dd9a8bf002c4b19eae6348909b61befdd30a4
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.710127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.710188    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.710348    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.710416    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.710395006 +0000 UTC m=+6.008746374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.710480    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:37.710896 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.710518    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.710507283 +0000 UTC m=+6.008858646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:37.912503 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:37.912415    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:37.912673 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.912607    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:37.912673 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.912631    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:37.912673 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.912648    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:37.912875 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:37.912711    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.912690422 +0000 UTC m=+6.211041778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:38.127526 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.127450    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:36 +0000 UTC" deadline="2028-02-24 22:59:10.404748201 +0000 UTC"
Jul 10 18:58:38.127526 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.127525    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14260h0m32.277228376s"
Jul 10 18:58:38.159777 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.159700    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:38.271851 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.271780    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:38.272001 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:38.271908    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:38.272352 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.272330    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:38.272458 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:38.272430    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:38.295468 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.295392    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerStarted","Data":"18e6703f6e685a60572f7a69ca81c9889c3d9470583a8dd6a86e1af8bacf43d8"}
Jul 10 18:58:38.309564 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.308922    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal" event={"ID":"1d70378dbc438c3228b7eaf66f8d5709","Type":"ContainerStarted","Data":"5f1a794534313a5178c7ef52b760c18bc693656a5fe36ead1e7e987f635bd3e2"}
Jul 10 18:58:38.321428 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.321399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsw4k" event={"ID":"9b103d35-f097-45f6-906b-11016846795f","Type":"ContainerStarted","Data":"7d309d384ad3b2cf8016782c419d54abb9eda888ce7743c11b4c5ef66f6c1976"}
Jul 10 18:58:38.333421 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.333354    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tz7r4" event={"ID":"d42dd62c-6e03-4e8d-bc39-3021e33abd05","Type":"ContainerStarted","Data":"43549396d242729151064efd272f459afa70995ae3497d72513d3052d8fa043d"}
Jul 10 18:58:38.343456 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.343353    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9755" event={"ID":"bb296e4e-b237-491f-9d54-f4363d7cfdcb","Type":"ContainerStarted","Data":"0c410ab53a705efb0fc70355d369f6b202b972d1292dc8c638e1cfd0bac5e0a9"}
Jul 10 18:58:38.353642 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.353567    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jmnrl" event={"ID":"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422","Type":"ContainerStarted","Data":"bebd8dd75abd40f0ad3c4fe71482210ac0430c44104ed632e891c89fc4d0ac99"}
Jul 10 18:58:38.359264 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.359238    2550 generic.go:356] "Generic (PLEG): container finished" podID="cfc1b2a120fe281acd7e52132f4adecf" containerID="4fddc43d8655e4173d0f9581b08e2f142ee3314b77e826ac67223ca32d2630ca" exitCode=0
Jul 10 18:58:38.359376 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.359313    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal" event={"ID":"cfc1b2a120fe281acd7e52132f4adecf","Type":"ContainerDied","Data":"4fddc43d8655e4173d0f9581b08e2f142ee3314b77e826ac67223ca32d2630ca"}
Jul 10 18:58:38.364080 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.364054    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pfg5n" event={"ID":"be349af5-8724-4dd3-94ed-7ac76841433e","Type":"ContainerStarted","Data":"bb9a08c5990ecc0089026eaa3f25a3766933632b9e97ea3715c674cf77534380"}
Jul 10 18:58:38.374528 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.374404    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"d76d54411a62a25524b83578e12dd9a8bf002c4b19eae6348909b61befdd30a4"}
Jul 10 18:58:38.376918 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.375925    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-20.ec2.internal" podStartSLOduration=3.37590788 podStartE2EDuration="3.37590788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:38.325769267 +0000 UTC m=+4.624120637" watchObservedRunningTime="2026-07-10 18:58:38.37590788 +0000 UTC m=+4.674259249"
Jul 10 18:58:38.382265 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.382240    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j" event={"ID":"83d48c51-3584-4f34-bd45-8235ae7c2e38","Type":"ContainerStarted","Data":"ebb4c84bb802e2c1d554516d7f6b4059fa8fe0d904e4e5fe0db90bfa7692ef15"}
Jul 10 18:58:38.386168 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:38.386144    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qbhw" event={"ID":"39adceed-d61c-41d6-abf8-2c341e30d3f9","Type":"ContainerStarted","Data":"c6cb72d9d4bec2db392b2e3b3a0deac23522fd0aebbc4fe6e7ad926919c12e73"}
Jul 10 18:58:39.270087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:39.270050    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:39.270568 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.270200    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:39.400966 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:39.400930    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal" event={"ID":"cfc1b2a120fe281acd7e52132f4adecf","Type":"ContainerStarted","Data":"b1eaabf6a0cb77bca9f9d1ef38d65cd77ba35b5735a016658e493472bb968e52"}
Jul 10 18:58:39.728377 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:39.728281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:39.728554 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:39.728405    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:39.728554 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.728532    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:39.728665 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.728596    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:43.728576867 +0000 UTC m=+10.026928238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:39.728758 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.728672    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:39.728758 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.728727    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:58:43.728694045 +0000 UTC m=+10.027045395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:39.930917 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:39.930201    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:39.930917 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.930387    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:39.930917 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.930413    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:39.930917 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.930426    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:39.930917 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:39.930485    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:43.930467548 +0000 UTC m=+10.228818900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:40.272369 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:40.272340    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:40.272819 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:40.272461    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:40.272819 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:40.272515    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:40.272819 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:40.272579    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:41.270063 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:41.270029    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:41.270254 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:41.270157    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:42.270648 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:42.270567    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:42.270648 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:42.270587    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:42.271155 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:42.270723    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:42.271155 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:42.270823    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:43.269837 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:43.269310    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:43.269837 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.269457    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:43.764352 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:43.764265    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:43.764352 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:43.764312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:43.764891 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.764431    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:43.764891 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.764500    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:58:51.764476585 +0000 UTC m=+18.062827939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:43.764996 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.764905    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:43.764996 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.764957    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:51.764941796 +0000 UTC m=+18.063293150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:43.966548 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:43.965908    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:43.966548 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.966090    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:43.966548 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.966114    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:43.966548 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.966127    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:43.966548 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:43.966186    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:51.966167159 +0000 UTC m=+18.264518525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:44.272695 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:44.272665    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:44.272887 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:44.272779    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:44.272887 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:44.272833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:44.272966 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:44.272906    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:45.270022 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:45.269986    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:45.270451 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:45.270168    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:46.274786 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:46.274339    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:46.274786 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:46.274347    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:46.274786 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:46.274469    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:46.274786 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:46.274524    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:47.270178 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:47.270142    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:47.270331 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:47.270263    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:48.269914 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:48.269827    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:48.270318 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:48.269841    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:48.270318 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:48.269952    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:48.270318 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:48.270067    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:49.269971 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:49.269929    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:49.270430 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:49.270086    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:50.269795 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:50.269756    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:50.269975 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:50.269890    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:50.269975 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:50.269922    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:50.270370 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:50.269986    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:51.270306 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:51.270273    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:51.270810 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:51.270413    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:51.823947 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:51.823919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:51.824058 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:51.823976    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:51.824058 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:51.824053    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:51.824132 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:51.824056    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:51.824132 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:51.824101    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:07.824087525 +0000 UTC m=+34.122438871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:51.824132 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:51.824113    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:59:07.824107069 +0000 UTC m=+34.122458416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:52.025641 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:52.025602    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:52.025816 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.025795    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:52.025876 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.025824    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:52.025876 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.025837    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:52.025958 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.025907    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:08.025886961 +0000 UTC m=+34.324238332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:52.270462 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:52.269940    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:52.270462 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.270084    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:52.270462 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:52.270133    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:52.270462 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:52.270261    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:53.270056 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:53.270017    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:53.270241 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:53.270140    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:54.272680 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:54.272646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:54.273142 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:54.272646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:54.273142 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:54.272769    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:54.273142 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:54.272818    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:55.269556 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:55.269515    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:55.269723 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:55.269661    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:56.271073 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.270880    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:56.271476 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:56.271184    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:56.271542 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.271507    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:56.271629 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:56.271602    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:56.444261 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.444157    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4qbhw" event={"ID":"39adceed-d61c-41d6-abf8-2c341e30d3f9","Type":"ContainerStarted","Data":"1b18f9669341562d782014ff1d64df434ec11d1d367cbbb1fe799d4c01f90909"}
Jul 10 18:58:56.446827 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.446794    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerStarted","Data":"00a8656a1b10878e1754e8258b6255d2469c02925bedbd02ee0837559bdb629d"}
Jul 10 18:58:56.451415 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.449618    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsw4k" event={"ID":"9b103d35-f097-45f6-906b-11016846795f","Type":"ContainerStarted","Data":"ceae12ba0f882492088f26c2f717d61810536d7e5569079fd5d8cfff99777286"}
Jul 10 18:58:56.454673 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.453133    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tz7r4" event={"ID":"d42dd62c-6e03-4e8d-bc39-3021e33abd05","Type":"ContainerStarted","Data":"caf2978bcea06e7a91e1bdc5859adb894646e62ad71d029994bdbb9c868680d1"}
Jul 10 18:58:56.456336 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.454766    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jmnrl" event={"ID":"a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422","Type":"ContainerStarted","Data":"201d9535c15c9fc897ee97963598fa2570ceca30b67b27e07500bab01119ddf5"}
Jul 10 18:58:56.461234 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.456997    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pfg5n" event={"ID":"be349af5-8724-4dd3-94ed-7ac76841433e","Type":"ContainerStarted","Data":"a2836d6a4c473c4d1006ad430025187e62739790d71961ab731ddae3c1db460b"}
Jul 10 18:58:56.461234 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.459366    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"2cbc377611a3b54f14050fbc93ad30bae069f3c4a580ff7936cfc7c6e88306e8"}
Jul 10 18:58:56.461234 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.461109    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j" event={"ID":"83d48c51-3584-4f34-bd45-8235ae7c2e38","Type":"ContainerStarted","Data":"55246bdfdc2673a514349fb1b657a5d875e2e5776f1664ff09f0cc6f2aee9184"}
Jul 10 18:58:56.468203 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.467705    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-20.ec2.internal" podStartSLOduration=21.467692554 podStartE2EDuration="21.467692554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:39.4193884 +0000 UTC m=+5.717739769" watchObservedRunningTime="2026-07-10 18:58:56.467692554 +0000 UTC m=+22.766043923"
Jul 10 18:58:56.468203 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.468058    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4qbhw" podStartSLOduration=3.488727231 podStartE2EDuration="22.468044375s" totalImagesPullingTime="18.979317144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.468004084 +0000 UTC m=+22.766355454" watchObservedRunningTime="2026-07-10 18:58:56.468044375 +0000 UTC m=+22.766395746"
Jul 10 18:58:56.553284 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.553242    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pfg5n" podStartSLOduration=3.63233154 podStartE2EDuration="22.553226821s" totalImagesPullingTime="18.920895281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.514774793 +0000 UTC m=+22.813126160" watchObservedRunningTime="2026-07-10 18:58:56.553226821 +0000 UTC m=+22.851578218"
Jul 10 18:58:56.591202 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.591151    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tz7r4" podStartSLOduration=8.100864957 podStartE2EDuration="22.591133868s" totalImagesPullingTime="14.490268911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.590785987 +0000 UTC m=+22.889137357" watchObservedRunningTime="2026-07-10 18:58:56.591133868 +0000 UTC m=+22.889485236"
Jul 10 18:58:56.591317 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:56.591280    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jmnrl" podStartSLOduration=3.66082367 podStartE2EDuration="22.591276227s" totalImagesPullingTime="18.930452557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.553823346 +0000 UTC m=+22.852174735" watchObservedRunningTime="2026-07-10 18:58:56.591276227 +0000 UTC m=+22.889627656"
Jul 10 18:58:57.269898 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.269869    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:57.270069 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:57.269995    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:57.464369 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.464326    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="00a8656a1b10878e1754e8258b6255d2469c02925bedbd02ee0837559bdb629d" exitCode=0
Jul 10 18:58:57.464369 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.464361    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"00a8656a1b10878e1754e8258b6255d2469c02925bedbd02ee0837559bdb629d"}
Jul 10 18:58:57.470990 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.470963    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"037a0364f375f35f562b88634ca78cbc4f2e43a7d7de6f4eb7af9e4148d1cec8"}
Jul 10 18:58:57.471122 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.471104    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"dfb98d4a295c55be6477579dd784629beaad440b5d9c4af51bebee58736b1295"}
Jul 10 18:58:57.471192 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.471163    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"faa36f4c0f328607eda58fa58aaa4363d74dca9ff4ec7aaaf03909de1f636f48"}
Jul 10 18:58:57.471249 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.471212    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"dd9faf6c32b89fae9e5a4ee47aff6e351242bb968708153a9c13cfdf88263086"}
Jul 10 18:58:57.471303 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.471270    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"eef9a30074ef4b64be2374077259283e66267a0270cb79092b660aab8e90847e"}
Jul 10 18:58:57.491945 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.491903    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zsw4k" podStartSLOduration=4.576578928 podStartE2EDuration="23.491890607s" totalImagesPullingTime="18.915311679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.626205834 +0000 UTC m=+22.924557205" watchObservedRunningTime="2026-07-10 18:58:57.491890607 +0000 UTC m=+23.790242013"
Jul 10 18:58:57.723064 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:57.723043    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:58:58.152606 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.152460    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:58:57.723061394Z","UUID":"df16cce8-3918-4758-846b-82e3d6c4d0f9","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:58:58.154240 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.154220    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:58:58.154362 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.154259    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:58:58.269934 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.269904    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:58:58.270101 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.269911    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:58:58.270101 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:58.270029    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:58:58.270209 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:58.270110    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:58:58.474909 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.474841    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9755" event={"ID":"bb296e4e-b237-491f-9d54-f4363d7cfdcb","Type":"ContainerStarted","Data":"e4ed2c5dcf5eb4ef8f030244743717407d21498803c85212794df18e89e0667d"}
Jul 10 18:58:58.477837 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.477805    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j" event={"ID":"83d48c51-3584-4f34-bd45-8235ae7c2e38","Type":"ContainerStarted","Data":"cdfde97716bae04b3a492d87a7f6a23b83c33b2cfc40fff4b0b9f188f4c03815"}
Jul 10 18:58:58.496321 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:58.496277    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k9755" podStartSLOduration=5.572472459 podStartE2EDuration="24.496259228s" totalImagesPullingTime="18.923786769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:58.496035249 +0000 UTC m=+24.794386619" watchObservedRunningTime="2026-07-10 18:58:58.496259228 +0000 UTC m=+24.794610599"
Jul 10 18:58:59.011447 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.011282    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:59.011911 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.011889    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:59.270060 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.269979    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:58:59.270219 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:58:59.270119    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:58:59.488694 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.488127    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"bc7818b0fcea56a135a395eb7e87076f91eccc6123481114aa1aa0fd1a5d0bd3"}
Jul 10 18:58:59.491053 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.491025    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j" event={"ID":"83d48c51-3584-4f34-bd45-8235ae7c2e38","Type":"ContainerStarted","Data":"28d76e28e1601a3863414feccf82de96e768b7eb115fc6ecf4be9fe66d7f165c"}
Jul 10 18:58:59.491414 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.491385    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:59.491967 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.491948    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tz7r4"
Jul 10 18:58:59.514965 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:58:59.514919    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlh6j" podStartSLOduration=4.239747956 podStartE2EDuration="25.514906651s" totalImagesPullingTime="21.275158695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.514758925 +0000 UTC m=+25.813110295" watchObservedRunningTime="2026-07-10 18:58:59.514906651 +0000 UTC m=+25.813258020"
Jul 10 18:59:00.269394 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:00.269359    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:00.269576 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:00.269364    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:00.269576 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:00.269476    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:00.269671 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:00.269573    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:01.269775 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:01.269726    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:01.270392 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:01.269878    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:02.270279 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.270029    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:02.270850 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.270076    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:02.270850 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:02.270383    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:02.270850 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:02.270420    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:02.500902 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.500868    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" event={"ID":"56678559-0a6b-42b5-b192-5d8bcd4a36ed","Type":"ContainerStarted","Data":"8fef82956dd1309f3a2bbae5737eb6de61c8550578b8951802b661c671b8931a"}
Jul 10 18:59:02.501168 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.501090    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:02.502661 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.502637    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="e0084bea7538f9479fd25a82e3bf06faaf0b5ceb77fa2919034c4793e735f514" exitCode=0
Jul 10 18:59:02.502768 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.502675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"e0084bea7538f9479fd25a82e3bf06faaf0b5ceb77fa2919034c4793e735f514"}
Jul 10 18:59:02.518543 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.518522    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:02.532021 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:02.531937    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9x886" podStartSLOduration=9.546857334 podStartE2EDuration="28.531922755s" totalImagesPullingTime="18.985065421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:02.530495864 +0000 UTC m=+28.828847259" watchObservedRunningTime="2026-07-10 18:59:02.531922755 +0000 UTC m=+28.830274124"
Jul 10 18:59:03.269877 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.269848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:03.270115 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:03.269964    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:03.505496 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.505471    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:03.506029 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.505540    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:03.521850 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.521774    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:03.789491 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.789162    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfhhk"]
Jul 10 18:59:03.789682 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.789518    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:03.789682 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:03.789622    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:03.789911 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.789848    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9gm9"]
Jul 10 18:59:03.789977 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.789958    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:03.790092 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:03.790057    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:03.790476 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.790458    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5mrgz"]
Jul 10 18:59:03.790561 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:03.790547    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:03.790673 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:03.790653    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:04.509129 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:04.509097    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="5c2ee0c7d40896c2580f6bae6e2c99f20222cb505b86e3665155bc9d72bf92bd" exitCode=0
Jul 10 18:59:04.509552 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:04.509181    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"5c2ee0c7d40896c2580f6bae6e2c99f20222cb505b86e3665155bc9d72bf92bd"}
Jul 10 18:59:05.269905 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:05.269876    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:05.270063 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:05.269879    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:05.270063 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:05.270008    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:05.270063 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:05.270020    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:05.270191 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:05.270096    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:05.270191 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:05.270165    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:06.515989 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:06.515957    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="0cc2c605feaf96d41e4e31300f2b030fba6fec80418e5727041c69f349243126" exitCode=0
Jul 10 18:59:06.516664 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:06.516000    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"0cc2c605feaf96d41e4e31300f2b030fba6fec80418e5727041c69f349243126"}
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:07.269570    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:07.269689    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.269694    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:07.269763    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.269861    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:07.270052 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.269968    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:07.840515 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:07.840472    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:07.841034 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:07.840531    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:07.841034 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.840654    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:59:07.841034 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.840665    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:59:07.841034 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.840719    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret podName:537fede7-1201-4688-9690-276e542b8f13 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:39.840704875 +0000 UTC m=+66.139056222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret") pod "global-pull-secret-syncer-w9gm9" (UID: "537fede7-1201-4688-9690-276e542b8f13") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:59:07.841034 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:07.840804    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs podName:bad76711-2921-4515-b2f4-26554d0783aa nodeName:}" failed. No retries permitted until 2026-07-10 18:59:39.840777106 +0000 UTC m=+66.139128467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs") pod "network-metrics-daemon-5mrgz" (UID: "bad76711-2921-4515-b2f4-26554d0783aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:59:08.042119 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:08.042006    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:08.042276 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:08.042136    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:59:08.042276 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:08.042158    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:59:08.042276 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:08.042170    2550 projected.go:196] Error preparing data for projected volume kube-api-access-nxflj for pod openshift-network-diagnostics/network-check-target-sfhhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:59:08.042276 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:08.042220    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj podName:93732cb3-2584-40f5-babc-d6e086652f69 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:40.042207972 +0000 UTC m=+66.340559320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxflj" (UniqueName: "kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj") pod "network-check-target-sfhhk" (UID: "93732cb3-2584-40f5-babc-d6e086652f69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:59:09.269664 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:09.269459    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:09.270146 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:09.269460    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:09.270146 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:09.269779    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9gm9" podUID="537fede7-1201-4688-9690-276e542b8f13"
Jul 10 18:59:09.270146 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:09.269459    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:09.270146 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:09.269854    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5mrgz" podUID="bad76711-2921-4515-b2f4-26554d0783aa"
Jul 10 18:59:09.270146 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:09.269913    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfhhk" podUID="93732cb3-2584-40f5-babc-d6e086652f69"
Jul 10 18:59:10.052005 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.051896    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-20.ec2.internal" event="NodeReady"
Jul 10 18:59:10.052163 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.052052    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:59:10.112758 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.112697    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gm5jl"]
Jul 10 18:59:10.116008 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.115987    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.119711 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.117071    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ww9lx"]
Jul 10 18:59:10.119711 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.119427    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99kdr\""
Jul 10 18:59:10.120242 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.120063    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:59:10.120242 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.120104    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:59:10.120880 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.120860    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.124471 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.124442    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:59:10.124580 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.124555    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:59:10.125077 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.125039    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2w9vx\""
Jul 10 18:59:10.125193 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.125135    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:59:10.132670 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.132634    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gm5jl"]
Jul 10 18:59:10.140444 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.140424    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ww9lx"]
Jul 10 18:59:10.150554 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.150535    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qslt6"]
Jul 10 18:59:10.153055 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.153034    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.155390 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.155369    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2qznj\""
Jul 10 18:59:10.156482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.156459    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:59:10.156570 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.156503    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:59:10.156758 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.156726    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:59:10.156845 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.156797    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:59:10.169465 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.169442    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qslt6"]
Jul 10 18:59:10.259791 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259759    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.259978 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259801    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.259978 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259827    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-api-access-mgjmd\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.259978 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae930abd-fa30-48d4-a391-403d4d3c3073-tmp-dir\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.259978 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jtl\" (UniqueName: \"kubernetes.io/projected/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-kube-api-access-t9jtl\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.260188 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.259992    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-crio-socket\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.260188 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.260040    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-cert\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.260188 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.260100    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae930abd-fa30-48d4-a391-403d4d3c3073-config-volume\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.260188 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.260139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.260382 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.260201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznwq\" (UniqueName: \"kubernetes.io/projected/ae930abd-fa30-48d4-a391-403d4d3c3073-kube-api-access-rznwq\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.260382 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.260242    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-data-volume\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.361438 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361358    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae930abd-fa30-48d4-a391-403d4d3c3073-tmp-dir\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.361438 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jtl\" (UniqueName: \"kubernetes.io/projected/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-kube-api-access-t9jtl\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.361438 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-crio-socket\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-cert\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361718    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-crio-socket\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361774    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae930abd-fa30-48d4-a391-403d4d3c3073-tmp-dir\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae930abd-fa30-48d4-a391-403d4d3c3073-config-volume\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361866    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznwq\" (UniqueName: \"kubernetes.io/projected/ae930abd-fa30-48d4-a391-403d4d3c3073-kube-api-access-rznwq\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361893    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-data-volume\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361926    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.361978    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-api-access-mgjmd\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362289    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.362318    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-data-volume\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362341    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls podName:ae930abd-fa30-48d4-a391-403d4d3c3073 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.862325936 +0000 UTC m=+37.160677282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls") pod "dns-default-gm5jl" (UID: "ae930abd-fa30-48d4-a391-403d4d3c3073") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.362356    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae930abd-fa30-48d4-a391-403d4d3c3073-config-volume\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362380    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:10.362508 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362453    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.862435944 +0000 UTC m=+37.160787294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:10.362782 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362520    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:10.362782 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.362547    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.862539303 +0000 UTC m=+37.160890650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:10.366227 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.366207    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-cert\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.373751 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.373705    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznwq\" (UniqueName: \"kubernetes.io/projected/ae930abd-fa30-48d4-a391-403d4d3c3073-kube-api-access-rznwq\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.373982 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.373963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jtl\" (UniqueName: \"kubernetes.io/projected/50b1ee8c-3513-412b-bce4-68e0dcfe1b79-kube-api-access-t9jtl\") pod \"ingress-canary-ww9lx\" (UID: \"50b1ee8c-3513-412b-bce4-68e0dcfe1b79\") " pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.374643 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.374624    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-api-access-mgjmd\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.437787 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.437745    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ww9lx"
Jul 10 18:59:10.649015 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.648988    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ww9lx"]
Jul 10 18:59:10.653489 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:10.653466    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b1ee8c_3513_412b_bce4_68e0dcfe1b79.slice/crio-8644752b33dea8e9fefa09c47898bdd4d81b4733a351acf82ea6713560722740 WatchSource:0}: Error finding container 8644752b33dea8e9fefa09c47898bdd4d81b4733a351acf82ea6713560722740: Status 404 returned error can't find the container with id 8644752b33dea8e9fefa09c47898bdd4d81b4733a351acf82ea6713560722740
Jul 10 18:59:10.864916 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.864825    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.864916 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.864882    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:10.864916 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:10.864902    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.864980    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.864997    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.865003    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.865050    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:11.865036741 +0000 UTC m=+38.163388088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.865062    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:11.865056483 +0000 UTC m=+38.163407830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:10.865157 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:10.865072    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls podName:ae930abd-fa30-48d4-a391-403d4d3c3073 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:11.86506582 +0000 UTC m=+38.163417168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls") pod "dns-default-gm5jl" (UID: "ae930abd-fa30-48d4-a391-403d4d3c3073") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:11.270163 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.270126    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:11.270340 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.270240    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:11.270340 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.270277    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:11.272957 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.272934    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:59:11.273293 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.273268    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:59:11.273384 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.273295    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvd26\""
Jul 10 18:59:11.274006 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.273986    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k2x45\""
Jul 10 18:59:11.274094 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.274016    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:59:11.274193 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.274179    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:59:11.530828 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.530584    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="584db8e403b2310b7718de13dfd2a0feb0291028dbc82197d382fc0a48c9e750" exitCode=0
Jul 10 18:59:11.531173 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.530625    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"584db8e403b2310b7718de13dfd2a0feb0291028dbc82197d382fc0a48c9e750"}
Jul 10 18:59:11.532560 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.532528    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ww9lx" event={"ID":"50b1ee8c-3513-412b-bce4-68e0dcfe1b79","Type":"ContainerStarted","Data":"8644752b33dea8e9fefa09c47898bdd4d81b4733a351acf82ea6713560722740"}
Jul 10 18:59:11.877463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.877370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:11.877463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.877435    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:11.877521    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877531    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877604    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.877587164 +0000 UTC m=+40.175938511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877607    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877531    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877656    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.877641428 +0000 UTC m=+40.175992775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:11.877701 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:11.877676    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls podName:ae930abd-fa30-48d4-a391-403d4d3c3073 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.877662615 +0000 UTC m=+40.176013979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls") pod "dns-default-gm5jl" (UID: "ae930abd-fa30-48d4-a391-403d4d3c3073") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:12.025647 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:12.025621    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jmnrl_a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422/dns-node-resolver/0.log"
Jul 10 18:59:12.538482 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:12.538446    2550 generic.go:356] "Generic (PLEG): container finished" podID="5691beb4-1cf8-4df5-b6a4-f11bb372c145" containerID="ef1ac9b88f308892941e490f260940be675a1bea7b68eee555a75de86962c166" exitCode=0
Jul 10 18:59:12.539006 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:12.538510    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerDied","Data":"ef1ac9b88f308892941e490f260940be675a1bea7b68eee555a75de86962c166"}
Jul 10 18:59:13.005955 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.005927    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsw4k_9b103d35-f097-45f6-906b-11016846795f/node-ca/0.log"
Jul 10 18:59:13.344680 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.344646    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"]
Jul 10 18:59:13.356963 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.356926    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"]
Jul 10 18:59:13.357095 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.357040    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.359463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.359432    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 10 18:59:13.359776 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.359758    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fbg6z\""
Jul 10 18:59:13.359972 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.359958    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 10 18:59:13.491912 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.491881    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.492066 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.491919    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.544605 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.544569    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ww9lx" event={"ID":"50b1ee8c-3513-412b-bce4-68e0dcfe1b79","Type":"ContainerStarted","Data":"43c6a238908547bf015d9612b30a24cfe7e00578a736c0df5cdd5fb5b476b958"}
Jul 10 18:59:13.547938 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.547912    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" event={"ID":"5691beb4-1cf8-4df5-b6a4-f11bb372c145","Type":"ContainerStarted","Data":"df7d2ec2f57676d24fc2b04668d3743a925062b11c198c6b48156b2a092ea368"}
Jul 10 18:59:13.564012 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.563964    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ww9lx" podStartSLOduration=1.111972585 podStartE2EDuration="3.563949975s" totalImagesPullingTime="2.45197739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:13.56372521 +0000 UTC m=+39.862076580" watchObservedRunningTime="2026-07-10 18:59:13.563949975 +0000 UTC m=+39.862301321"
Jul 10 18:59:13.588540 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.588494    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pzkb7" podStartSLOduration=10.5357095 podStartE2EDuration="39.588480654s" totalImagesPullingTime="29.052771154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:13.587018105 +0000 UTC m=+39.885369510" watchObservedRunningTime="2026-07-10 18:59:13.588480654 +0000 UTC m=+39.886832057"
Jul 10 18:59:13.593251 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.593226    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.593368 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.593263    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.593412 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.593384    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 10 18:59:13.593455 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.593450    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert podName:fe7f67ed-0c28-4b62-8b30-2add5ab2412e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:14.093435385 +0000 UTC m=+40.391786732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-2wg4d" (UID: "fe7f67ed-0c28-4b62-8b30-2add5ab2412e") : secret "networking-console-plugin-cert" not found
Jul 10 18:59:13.593869 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.593853    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:13.897040 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.897017    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:13.897185 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.897065    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:13.897185 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:13.897087    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:13.897185 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897149    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:13.897185 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897175    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:13.897185 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897175    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:13.897332 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897220    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:17.897208148 +0000 UTC m=+44.195559496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:13.897332 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897233    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:17.897226755 +0000 UTC m=+44.195578102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:13.897332 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:13.897243    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls podName:ae930abd-fa30-48d4-a391-403d4d3c3073 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:17.897236768 +0000 UTC m=+44.195588115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls") pod "dns-default-gm5jl" (UID: "ae930abd-fa30-48d4-a391-403d4d3c3073") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:14.099183 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:14.099141    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:14.099335 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:14.099286    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 10 18:59:14.099371 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:14.099350    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert podName:fe7f67ed-0c28-4b62-8b30-2add5ab2412e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:15.099333506 +0000 UTC m=+41.397684852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-2wg4d" (UID: "fe7f67ed-0c28-4b62-8b30-2add5ab2412e") : secret "networking-console-plugin-cert" not found
Jul 10 18:59:15.104498 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.104454    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:15.104928 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:15.104609    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 10 18:59:15.104928 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:15.104686    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert podName:fe7f67ed-0c28-4b62-8b30-2add5ab2412e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:17.104670511 +0000 UTC m=+43.403021858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-2wg4d" (UID: "fe7f67ed-0c28-4b62-8b30-2add5ab2412e") : secret "networking-console-plugin-cert" not found
Jul 10 18:59:15.167852 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.167822    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-cf97t"]
Jul 10 18:59:15.198403 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.198372    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-cf97t"]
Jul 10 18:59:15.198520 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.198466    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.201133 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.201109    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 10 18:59:15.202185 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.202165    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 10 18:59:15.202297 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.202214    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-whqtf\""
Jul 10 18:59:15.202358 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.202299    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 10 18:59:15.202408 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.202376    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 10 18:59:15.204935 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.204918    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-cabundle\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.205031 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.204961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-key\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.205087 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.205075    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rrw\" (UniqueName: \"kubernetes.io/projected/03a93db9-abb6-4e1b-a52b-e78a45256f0a-kube-api-access-v9rrw\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.305810 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.305773    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rrw\" (UniqueName: \"kubernetes.io/projected/03a93db9-abb6-4e1b-a52b-e78a45256f0a-kube-api-access-v9rrw\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.305985 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.305827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-cabundle\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.305985 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.305853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-key\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.306586 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.306563    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-cabundle\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.309353 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.309333    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03a93db9-abb6-4e1b-a52b-e78a45256f0a-signing-key\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.315014 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.314997    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rrw\" (UniqueName: \"kubernetes.io/projected/03a93db9-abb6-4e1b-a52b-e78a45256f0a-kube-api-access-v9rrw\") pod \"service-ca-79f68b9f4f-cf97t\" (UID: \"03a93db9-abb6-4e1b-a52b-e78a45256f0a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.507890 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.507842    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t"
Jul 10 18:59:15.654317 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:15.654281    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-cf97t"]
Jul 10 18:59:15.659257 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:15.659232    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a93db9_abb6_4e1b_a52b_e78a45256f0a.slice/crio-cef600584e983ff72bd5d82cdcd15a1f1699c0e12c2ed24c321d91c7b9f40a3d WatchSource:0}: Error finding container cef600584e983ff72bd5d82cdcd15a1f1699c0e12c2ed24c321d91c7b9f40a3d: Status 404 returned error can't find the container with id cef600584e983ff72bd5d82cdcd15a1f1699c0e12c2ed24c321d91c7b9f40a3d
Jul 10 18:59:16.556722 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:16.556681    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t" event={"ID":"03a93db9-abb6-4e1b-a52b-e78a45256f0a","Type":"ContainerStarted","Data":"cef600584e983ff72bd5d82cdcd15a1f1699c0e12c2ed24c321d91c7b9f40a3d"}
Jul 10 18:59:17.120964 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:17.120916    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:17.121129 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.121096    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 10 18:59:17.121193 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.121181    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert podName:fe7f67ed-0c28-4b62-8b30-2add5ab2412e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:21.121163541 +0000 UTC m=+47.419514894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-2wg4d" (UID: "fe7f67ed-0c28-4b62-8b30-2add5ab2412e") : secret "networking-console-plugin-cert" not found
Jul 10 18:59:17.927606 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:17.927570    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:17.927626    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:17.927648    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927705    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927729    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927780    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927796    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:25.92778033 +0000 UTC m=+52.226131677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927819    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:25.927806523 +0000 UTC m=+52.226157870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:17.927981 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:17.927835    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls podName:ae930abd-fa30-48d4-a391-403d4d3c3073 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:25.927827218 +0000 UTC m=+52.226178565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls") pod "dns-default-gm5jl" (UID: "ae930abd-fa30-48d4-a391-403d4d3c3073") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:18.563288 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:18.563255    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t" event={"ID":"03a93db9-abb6-4e1b-a52b-e78a45256f0a","Type":"ContainerStarted","Data":"153bd3a3c2ed6d112da47789253acb4f4ebe223a9e9b5cee154a23669d42bbaa"}
Jul 10 18:59:18.591377 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:18.591336    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-cf97t" podStartSLOduration=1.23142029 podStartE2EDuration="3.59132239s" totalImagesPullingTime="2.3599021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:18.591085111 +0000 UTC m=+44.889436480" watchObservedRunningTime="2026-07-10 18:59:18.59132239 +0000 UTC m=+44.889673758"
Jul 10 18:59:21.153164 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:21.153129    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:21.153631 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:21.153230    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 10 18:59:21.153631 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:21.153283    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert podName:fe7f67ed-0c28-4b62-8b30-2add5ab2412e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:29.153268868 +0000 UTC m=+55.451620216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-2wg4d" (UID: "fe7f67ed-0c28-4b62-8b30-2add5ab2412e") : secret "networking-console-plugin-cert" not found
Jul 10 18:59:25.992627 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:25.992591    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:25.993136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:25.992661    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:25.993136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:25.992691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:25.993136 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:25.992868    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:25.993136 ip-10-0-139-20 kubenswrapper[2550]: E0710 18:59:25.992943    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls podName:4867a0c0-f343-4ff2-bdf9-e4dbc9be4761 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.992923452 +0000 UTC m=+68.291274813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qslt6" (UID: "4867a0c0-f343-4ff2-bdf9-e4dbc9be4761") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:25.993136 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:25.993128    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:25.995156 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:25.995136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae930abd-fa30-48d4-a391-403d4d3c3073-metrics-tls\") pod \"dns-default-gm5jl\" (UID: \"ae930abd-fa30-48d4-a391-403d4d3c3073\") " pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:26.030596 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:26.030565    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:26.191423 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:26.190994    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gm5jl"]
Jul 10 18:59:26.196043 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:26.196014    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae930abd_fa30_48d4_a391_403d4d3c3073.slice/crio-1776e8e69663cbf8b8afd8e570bc5f94357717e0f857b04925ab039105f748ff WatchSource:0}: Error finding container 1776e8e69663cbf8b8afd8e570bc5f94357717e0f857b04925ab039105f748ff: Status 404 returned error can't find the container with id 1776e8e69663cbf8b8afd8e570bc5f94357717e0f857b04925ab039105f748ff
Jul 10 18:59:26.585309 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:26.585272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm5jl" event={"ID":"ae930abd-fa30-48d4-a391-403d4d3c3073","Type":"ContainerStarted","Data":"1776e8e69663cbf8b8afd8e570bc5f94357717e0f857b04925ab039105f748ff"}
Jul 10 18:59:29.217991 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.217956    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:29.220357 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.220335    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7f67ed-0c28-4b62-8b30-2add5ab2412e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2wg4d\" (UID: \"fe7f67ed-0c28-4b62-8b30-2add5ab2412e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:29.268546 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.268510    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"
Jul 10 18:59:29.395579 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.395545    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2wg4d"]
Jul 10 18:59:29.398930 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:29.398908    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7f67ed_0c28_4b62_8b30_2add5ab2412e.slice/crio-61a659c4c71a372316fea2e8c0d1ea6250a1603ae52730dcb788e6a7f7cd4ee3 WatchSource:0}: Error finding container 61a659c4c71a372316fea2e8c0d1ea6250a1603ae52730dcb788e6a7f7cd4ee3: Status 404 returned error can't find the container with id 61a659c4c71a372316fea2e8c0d1ea6250a1603ae52730dcb788e6a7f7cd4ee3
Jul 10 18:59:29.593383 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.593307    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d" event={"ID":"fe7f67ed-0c28-4b62-8b30-2add5ab2412e","Type":"ContainerStarted","Data":"61a659c4c71a372316fea2e8c0d1ea6250a1603ae52730dcb788e6a7f7cd4ee3"}
Jul 10 18:59:29.595078 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.595050    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm5jl" event={"ID":"ae930abd-fa30-48d4-a391-403d4d3c3073","Type":"ContainerStarted","Data":"fccfc48a05026d6c9fdc6f93ac57a98dc2f6f8af7980267523822fec9194b2b8"}
Jul 10 18:59:29.595206 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.595094    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm5jl" event={"ID":"ae930abd-fa30-48d4-a391-403d4d3c3073","Type":"ContainerStarted","Data":"e1965fec80cf080d19c1ea968cd128bb07fbbc354fe059d1acac64bb9c3a91b3"}
Jul 10 18:59:29.595206 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.595190    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:29.620366 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:29.620326    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gm5jl" podStartSLOduration=17.2844199 podStartE2EDuration="19.620313092s" totalImagesPullingTime="2.335893192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:29.618847252 +0000 UTC m=+55.917198621" watchObservedRunningTime="2026-07-10 18:59:29.620313092 +0000 UTC m=+55.918664460"
Jul 10 18:59:31.603592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:31.603553    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d" event={"ID":"fe7f67ed-0c28-4b62-8b30-2add5ab2412e","Type":"ContainerStarted","Data":"2bd1400c374280480abb44618a1422e9e2ece1a233ba606b9910b44f25840d64"}
Jul 10 18:59:31.622195 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:31.622148    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-2wg4d" podStartSLOduration=17.483216911 podStartE2EDuration="18.622135237s" totalImagesPullingTime="1.138918326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:31.621794807 +0000 UTC m=+57.920146178" watchObservedRunningTime="2026-07-10 18:59:31.622135237 +0000 UTC m=+57.920486605"
Jul 10 18:59:35.526874 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:35.526842    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x886"
Jul 10 18:59:38.883143 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.883107    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"]
Jul 10 18:59:38.885838 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.885805    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:38.892405 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.892381    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:59:38.892615 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.892594    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:59:38.893672 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.893616    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 10 18:59:38.893819 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.893764    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jhwf9\""
Jul 10 18:59:38.895335 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.895315    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"]
Jul 10 18:59:38.897351 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.897332    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.898124 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.898106    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:59:38.900416 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.900398    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 18:59:38.900416 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.900412    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 18:59:38.900555 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.900402    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 18:59:38.900662 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.900650    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 18:59:38.912279 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.911904    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"]
Jul 10 18:59:38.917677 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.917652    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"]
Jul 10 18:59:38.992196 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992166    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvwg\" (UniqueName: \"kubernetes.io/projected/019338f0-e2bc-46a9-92d6-de03870cef49-kube-api-access-dcvwg\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:38.992345 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.992345 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc72h\" (UniqueName: \"kubernetes.io/projected/9527dbd0-ff4c-4c05-a0da-91bdb5777652-kube-api-access-bc72h\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.992418 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.992451 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992424    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9527dbd0-ff4c-4c05-a0da-91bdb5777652-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.992515 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992500    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/019338f0-e2bc-46a9-92d6-de03870cef49-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:38.992547 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:38.992581 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:38.992563    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093524 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093495    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvwg\" (UniqueName: \"kubernetes.io/projected/019338f0-e2bc-46a9-92d6-de03870cef49-kube-api-access-dcvwg\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:39.093697 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093697 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093559    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc72h\" (UniqueName: \"kubernetes.io/projected/9527dbd0-ff4c-4c05-a0da-91bdb5777652-kube-api-access-bc72h\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093697 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093649    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093697 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093681    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9527dbd0-ff4c-4c05-a0da-91bdb5777652-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093938 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/019338f0-e2bc-46a9-92d6-de03870cef49-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:39.093938 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093804    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.093938 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.093831    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.094629 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.094582    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9527dbd0-ff4c-4c05-a0da-91bdb5777652-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.096520 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.096489    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/019338f0-e2bc-46a9-92d6-de03870cef49-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:39.096793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.096772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.096891 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.096795    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.096960 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.096884    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-ca\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.097041 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.097020    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9527dbd0-ff4c-4c05-a0da-91bdb5777652-hub\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.111962 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.111939    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc72h\" (UniqueName: \"kubernetes.io/projected/9527dbd0-ff4c-4c05-a0da-91bdb5777652-kube-api-access-bc72h\") pod \"cluster-proxy-proxy-agent-5f89556d47-hns9x\" (UID: \"9527dbd0-ff4c-4c05-a0da-91bdb5777652\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.114560 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.114538    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvwg\" (UniqueName: \"kubernetes.io/projected/019338f0-e2bc-46a9-92d6-de03870cef49-kube-api-access-dcvwg\") pod \"managed-serviceaccount-addon-agent-7d459dbcf6-qzztq\" (UID: \"019338f0-e2bc-46a9-92d6-de03870cef49\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:39.201998 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.201964    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"
Jul 10 18:59:39.214617 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.214588    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"
Jul 10 18:59:39.342188 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.342157    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq"]
Jul 10 18:59:39.345582 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:39.345541    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod019338f0_e2bc_46a9_92d6_de03870cef49.slice/crio-a4701287e4b5ada0d7b7b23748005a14bb2d169c9de357e9e132e1f01adfdbd2 WatchSource:0}: Error finding container a4701287e4b5ada0d7b7b23748005a14bb2d169c9de357e9e132e1f01adfdbd2: Status 404 returned error can't find the container with id a4701287e4b5ada0d7b7b23748005a14bb2d169c9de357e9e132e1f01adfdbd2
Jul 10 18:59:39.366838 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.366820    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x"]
Jul 10 18:59:39.368935 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:39.368913    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9527dbd0_ff4c_4c05_a0da_91bdb5777652.slice/crio-a1b2e8f16dfcf5b5b5a59f99666b1c4ea7d8e2407ec04fae39f9f128a849378c WatchSource:0}: Error finding container a1b2e8f16dfcf5b5b5a59f99666b1c4ea7d8e2407ec04fae39f9f128a849378c: Status 404 returned error can't find the container with id a1b2e8f16dfcf5b5b5a59f99666b1c4ea7d8e2407ec04fae39f9f128a849378c
Jul 10 18:59:39.602641 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.602561    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gm5jl"
Jul 10 18:59:39.623196 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.623156    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x" event={"ID":"9527dbd0-ff4c-4c05-a0da-91bdb5777652","Type":"ContainerStarted","Data":"a1b2e8f16dfcf5b5b5a59f99666b1c4ea7d8e2407ec04fae39f9f128a849378c"}
Jul 10 18:59:39.624211 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.624187    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq" event={"ID":"019338f0-e2bc-46a9-92d6-de03870cef49","Type":"ContainerStarted","Data":"a4701287e4b5ada0d7b7b23748005a14bb2d169c9de357e9e132e1f01adfdbd2"}
Jul 10 18:59:39.900730 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.900693    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:39.901162 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.900757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:39.903399 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.903382    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:59:39.903448 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.903410    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:59:39.913722 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.913694    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/537fede7-1201-4688-9690-276e542b8f13-original-pull-secret\") pod \"global-pull-secret-syncer-w9gm9\" (UID: \"537fede7-1201-4688-9690-276e542b8f13\") " pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:39.913840 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:39.913774    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad76711-2921-4515-b2f4-26554d0783aa-metrics-certs\") pod \"network-metrics-daemon-5mrgz\" (UID: \"bad76711-2921-4515-b2f4-26554d0783aa\") " pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:40.082028 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.081998    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9gm9"
Jul 10 18:59:40.097377 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.097353    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xvd26\""
Jul 10 18:59:40.102062 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.102038    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:40.105313 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.105290    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5mrgz"
Jul 10 18:59:40.105677 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.105658    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:59:40.115770 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.115723    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:59:40.126032 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.126011    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxflj\" (UniqueName: \"kubernetes.io/projected/93732cb3-2584-40f5-babc-d6e086652f69-kube-api-access-nxflj\") pod \"network-check-target-sfhhk\" (UID: \"93732cb3-2584-40f5-babc-d6e086652f69\") " pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:40.219065 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.219039    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9gm9"]
Jul 10 18:59:40.222208 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:40.222183    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537fede7_1201_4688_9690_276e542b8f13.slice/crio-4a1c7e68f5151dbbd6040ff36f116cbc9061ed377e3f85f4f70e5597837b087f WatchSource:0}: Error finding container 4a1c7e68f5151dbbd6040ff36f116cbc9061ed377e3f85f4f70e5597837b087f: Status 404 returned error can't find the container with id 4a1c7e68f5151dbbd6040ff36f116cbc9061ed377e3f85f4f70e5597837b087f
Jul 10 18:59:40.249266 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.249242    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5mrgz"]
Jul 10 18:59:40.251564 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:40.251542    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad76711_2921_4515_b2f4_26554d0783aa.slice/crio-487e0dd2f94314c5e5c5f147c66712f68f7f4cb0b064e63f767228c88635ecdf WatchSource:0}: Error finding container 487e0dd2f94314c5e5c5f147c66712f68f7f4cb0b064e63f767228c88635ecdf: Status 404 returned error can't find the container with id 487e0dd2f94314c5e5c5f147c66712f68f7f4cb0b064e63f767228c88635ecdf
Jul 10 18:59:40.391570 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.391535    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k2x45\""
Jul 10 18:59:40.399796 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.399770    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:40.558604 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.558572    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfhhk"]
Jul 10 18:59:40.562347 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:40.562317    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93732cb3_2584_40f5_babc_d6e086652f69.slice/crio-8d6fd8504873d8de7555f0b8dcceb6858cde88ee52504978d752d593fb5ae364 WatchSource:0}: Error finding container 8d6fd8504873d8de7555f0b8dcceb6858cde88ee52504978d752d593fb5ae364: Status 404 returned error can't find the container with id 8d6fd8504873d8de7555f0b8dcceb6858cde88ee52504978d752d593fb5ae364
Jul 10 18:59:40.630037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.629944    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5mrgz" event={"ID":"bad76711-2921-4515-b2f4-26554d0783aa","Type":"ContainerStarted","Data":"487e0dd2f94314c5e5c5f147c66712f68f7f4cb0b064e63f767228c88635ecdf"}
Jul 10 18:59:40.631650 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.631591    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9gm9" event={"ID":"537fede7-1201-4688-9690-276e542b8f13","Type":"ContainerStarted","Data":"4a1c7e68f5151dbbd6040ff36f116cbc9061ed377e3f85f4f70e5597837b087f"}
Jul 10 18:59:40.633312 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:40.633286    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfhhk" event={"ID":"93732cb3-2584-40f5-babc-d6e086652f69","Type":"ContainerStarted","Data":"8d6fd8504873d8de7555f0b8dcceb6858cde88ee52504978d752d593fb5ae364"}
Jul 10 18:59:42.022341 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:42.022292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:42.026300 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:42.026220    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4867a0c0-f343-4ff2-bdf9-e4dbc9be4761-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qslt6\" (UID: \"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761\") " pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:42.265249 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:42.264992    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2qznj\""
Jul 10 18:59:42.272867 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:42.272621    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qslt6"
Jul 10 18:59:47.410318 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.410165    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qslt6"]
Jul 10 18:59:47.414652 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:47.414629    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4867a0c0_f343_4ff2_bdf9_e4dbc9be4761.slice/crio-254dc71002dfd65d3ea5b878710cc343b2eafa451364e8e104e96c99579945b6 WatchSource:0}: Error finding container 254dc71002dfd65d3ea5b878710cc343b2eafa451364e8e104e96c99579945b6: Status 404 returned error can't find the container with id 254dc71002dfd65d3ea5b878710cc343b2eafa451364e8e104e96c99579945b6
Jul 10 18:59:47.668909 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.668869    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5mrgz" event={"ID":"bad76711-2921-4515-b2f4-26554d0783aa","Type":"ContainerStarted","Data":"85d4f1bad4e1eb2c62385de593b0660a703575ffe7a527ea8147ceb21d3e6c2d"}
Jul 10 18:59:47.669076 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.668942    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5mrgz" event={"ID":"bad76711-2921-4515-b2f4-26554d0783aa","Type":"ContainerStarted","Data":"521d326c5d01adf8ed051b3ae6c6afb2c3ba785a19518c7355c91a9308d9643e"}
Jul 10 18:59:47.670425 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.670399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9gm9" event={"ID":"537fede7-1201-4688-9690-276e542b8f13","Type":"ContainerStarted","Data":"97c4f63c93f63e00260df1c6798ce81a983beeef29ffff38d271e68f40fa0f94"}
Jul 10 18:59:47.672463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.672430    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x" event={"ID":"9527dbd0-ff4c-4c05-a0da-91bdb5777652","Type":"ContainerStarted","Data":"3805e2262d22d6b68b11bed94d79147b35eac87e28e186f3c7d1be0dd3168223"}
Jul 10 18:59:47.672463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.672464    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x" event={"ID":"9527dbd0-ff4c-4c05-a0da-91bdb5777652","Type":"ContainerStarted","Data":"1d058efc12ebddf442468a3190220e445616f1f26a43f18c88dff6e614357b52"}
Jul 10 18:59:47.672627 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.672481    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x" event={"ID":"9527dbd0-ff4c-4c05-a0da-91bdb5777652","Type":"ContainerStarted","Data":"406c53c7fa025c4a81d6abbf3d2bce196264c5d3a481752dc337fc01314ab9e2"}
Jul 10 18:59:47.674083 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.674062    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfhhk" event={"ID":"93732cb3-2584-40f5-babc-d6e086652f69","Type":"ContainerStarted","Data":"07fd9edaec9ca2c68878e80312437c58772c99701ef16c7d63c446fcbe5e374c"}
Jul 10 18:59:47.674169 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.674107    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 18:59:47.675240 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.675221    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qslt6" event={"ID":"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761","Type":"ContainerStarted","Data":"254dc71002dfd65d3ea5b878710cc343b2eafa451364e8e104e96c99579945b6"}
Jul 10 18:59:47.676509 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.676489    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq" event={"ID":"019338f0-e2bc-46a9-92d6-de03870cef49","Type":"ContainerStarted","Data":"0ea3b02f041ef063e4330e359a5fa81a4ee02ff8706c9d13e45d4130e7241aaa"}
Jul 10 18:59:47.690575 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.690536    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5mrgz" podStartSLOduration=66.715764882 podStartE2EDuration="1m13.690522837s" totalImagesPullingTime="6.974757955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.689799696 +0000 UTC m=+73.988151064" watchObservedRunningTime="2026-07-10 18:59:47.690522837 +0000 UTC m=+73.988874206"
Jul 10 18:59:47.710592 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.710543    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w9gm9" podStartSLOduration=66.685258911 podStartE2EDuration="1m13.710518153s" totalImagesPullingTime="7.025259242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.710411367 +0000 UTC m=+74.008762736" watchObservedRunningTime="2026-07-10 18:59:47.710518153 +0000 UTC m=+74.008869518"
Jul 10 18:59:47.741181 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.741098    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sfhhk" podStartSLOduration=67.058576845 podStartE2EDuration="1m13.741082498s" totalImagesPullingTime="6.682505653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.740274405 +0000 UTC m=+74.038625771" watchObservedRunningTime="2026-07-10 18:59:47.741082498 +0000 UTC m=+74.039433866"
Jul 10 18:59:47.810337 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:47.809937    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d459dbcf6-qzztq" podStartSLOduration=1.925366098 podStartE2EDuration="9.809892084s" totalImagesPullingTime="7.884525986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.764719556 +0000 UTC m=+74.063070927" watchObservedRunningTime="2026-07-10 18:59:47.809892084 +0000 UTC m=+74.108243454"
Jul 10 18:59:49.646998 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.646927    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f89556d47-hns9x" podStartSLOduration=3.768426582 podStartE2EDuration="11.646907585s" totalImagesPullingTime="7.878481003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.811631131 +0000 UTC m=+74.109982500" watchObservedRunningTime="2026-07-10 18:59:49.646907585 +0000 UTC m=+75.945258954"
Jul 10 18:59:49.647889 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.647867    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8lfbz"]
Jul 10 18:59:49.660094 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.660061    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.667711 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.667488    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:59:49.667711 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.667507    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:59:49.667711 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.667522    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:59:49.668753 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.668502    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j22gr\""
Jul 10 18:59:49.668753 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.668521    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:59:49.668753 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.668522    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:59:49.668753 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.668561    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:59:49.786257 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786220    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-root\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786426 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786270    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdbf\" (UniqueName: \"kubernetes.io/projected/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-kube-api-access-4bdbf\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786426 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786373    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-sys\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786426 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786410    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786578 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786436    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786578 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786492    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-tls\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786578 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786515    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-wtmp\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786578 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786547    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-metrics-client-ca\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.786754 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.786622    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-textfile\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887792 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887754    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-textfile\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887823    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-root\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdbf\" (UniqueName: \"kubernetes.io/projected/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-kube-api-access-4bdbf\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887888    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-sys\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887913    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887944    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.887976 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-tls\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888273 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888002    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-wtmp\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888273 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888042    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-metrics-client-ca\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888273 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888062    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-textfile\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888418 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.887972    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-root\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888418 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888355    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-sys\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888511 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888483    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-wtmp\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888547 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888534    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.888882 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.888861    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-metrics-client-ca\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.891047 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.891024    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.891191 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.891071    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-node-exporter-tls\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.897365 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.897306    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdbf\" (UniqueName: \"kubernetes.io/projected/01d4242f-5edc-40ed-982f-3b8ce5e92e0a-kube-api-access-4bdbf\") pod \"node-exporter-8lfbz\" (UID: \"01d4242f-5edc-40ed-982f-3b8ce5e92e0a\") " pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:49.971423 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:49.971384    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lfbz"
Jul 10 18:59:51.034171 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:51.034128    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d4242f_5edc_40ed_982f_3b8ce5e92e0a.slice/crio-075a47c45647b2c789e7493af05101c844fbe9775930a9ffded00a95b08241ef WatchSource:0}: Error finding container 075a47c45647b2c789e7493af05101c844fbe9775930a9ffded00a95b08241ef: Status 404 returned error can't find the container with id 075a47c45647b2c789e7493af05101c844fbe9775930a9ffded00a95b08241ef
Jul 10 18:59:51.691750 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:51.691702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lfbz" event={"ID":"01d4242f-5edc-40ed-982f-3b8ce5e92e0a","Type":"ContainerStarted","Data":"075a47c45647b2c789e7493af05101c844fbe9775930a9ffded00a95b08241ef"}
Jul 10 18:59:51.693222 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:51.693193    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qslt6" event={"ID":"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761","Type":"ContainerStarted","Data":"66dc6ff872e1be689b99d706a151ac0100a7934b610ece36df8855e0cd670dd9"}
Jul 10 18:59:52.644358 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.644322    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"]
Jul 10 18:59:52.649971 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.649946    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.653274 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.653251    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mjqpg\""
Jul 10 18:59:52.653853 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.653826    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 10 18:59:52.654037 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.654023    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 10 18:59:52.654104 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.654074    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 10 18:59:52.654258 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.654234    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 10 18:59:52.654387 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.654368    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 10 18:59:52.655022 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.655005    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-291268023pno8\""
Jul 10 18:59:52.695338 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.695304    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"]
Jul 10 18:59:52.700884 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.700849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qslt6" event={"ID":"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761","Type":"ContainerStarted","Data":"359aef13db3694f5c4b37215e13b3751a91045ae66833a2f536056e3df0e8a96"}
Jul 10 18:59:52.702969 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.702942    2550 generic.go:356] "Generic (PLEG): container finished" podID="01d4242f-5edc-40ed-982f-3b8ce5e92e0a" containerID="3e37ae7b7d27083bcac6a738d9f2d5f664a2d356946f78671c47ff806831777b" exitCode=0
Jul 10 18:59:52.703091 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.703015    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lfbz" event={"ID":"01d4242f-5edc-40ed-982f-3b8ce5e92e0a","Type":"ContainerDied","Data":"3e37ae7b7d27083bcac6a738d9f2d5f664a2d356946f78671c47ff806831777b"}
Jul 10 18:59:52.810880 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-metrics-client-ca\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.810880 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810794    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.810880 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.811106 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810910    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.811106 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.811106 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.810990    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dtt\" (UniqueName: \"kubernetes.io/projected/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-kube-api-access-v4dtt\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.811106 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.811050    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.811106 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.811080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-grpc-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912093 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912018    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-metrics-client-ca\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912093 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912070    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912093 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912101    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912361 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912139    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912361 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912361 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dtt\" (UniqueName: \"kubernetes.io/projected/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-kube-api-access-v4dtt\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912361 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912255    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912361 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912284    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-grpc-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.912927 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.912893    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-metrics-client-ca\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.915463 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.915429    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.915589 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.915556    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-grpc-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.915706 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.915683    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.915798 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.915693    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.915887 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.915866    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.917328 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.917310    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-secret-thanos-querier-tls\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.923793 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.923774    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dtt\" (UniqueName: \"kubernetes.io/projected/5d184c2b-5f86-40f4-8b51-0c94bccf42a5-kube-api-access-v4dtt\") pod \"thanos-querier-84f4cb5799-vd4h6\" (UID: \"5d184c2b-5f86-40f4-8b51-0c94bccf42a5\") " pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:52.963350 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:52.963328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:53.163377 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:53.163298    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"]
Jul 10 18:59:53.166679 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:53.166652    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d184c2b_5f86_40f4_8b51_0c94bccf42a5.slice/crio-cf39eb78b2bf6c492e64113039e766ad074d885643f5d1a163fbd6b4e2e09da0 WatchSource:0}: Error finding container cf39eb78b2bf6c492e64113039e766ad074d885643f5d1a163fbd6b4e2e09da0: Status 404 returned error can't find the container with id cf39eb78b2bf6c492e64113039e766ad074d885643f5d1a163fbd6b4e2e09da0
Jul 10 18:59:53.708969 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:53.708927    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lfbz" event={"ID":"01d4242f-5edc-40ed-982f-3b8ce5e92e0a","Type":"ContainerStarted","Data":"0f20ea2126606e335ae4fcc6fcc9532b14b818fd01174d31d9b702880446f710"}
Jul 10 18:59:53.709388 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:53.708990    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lfbz" event={"ID":"01d4242f-5edc-40ed-982f-3b8ce5e92e0a","Type":"ContainerStarted","Data":"3b565b9cddc33a5ecea57bc5df9b026df0506062bb7c4b545df944d71e22c174"}
Jul 10 18:59:53.710302 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:53.710273    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"cf39eb78b2bf6c492e64113039e766ad074d885643f5d1a163fbd6b4e2e09da0"}
Jul 10 18:59:53.738938 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:53.738882    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8lfbz" podStartSLOduration=3.7752258469999997 podStartE2EDuration="4.738866918s" totalImagesPullingTime="894.556198ms" totalInitContainerRuntime="69.084873ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:53.737357711 +0000 UTC m=+80.035709082" watchObservedRunningTime="2026-07-10 18:59:53.738866918 +0000 UTC m=+80.037218287"
Jul 10 18:59:54.265867 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.265836    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"]
Jul 10 18:59:54.275645 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.275239    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.278170 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.278144    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 10 18:59:54.278279 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.278259    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vbrb4\""
Jul 10 18:59:54.293602 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.293570    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"]
Jul 10 18:59:54.426233 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.426190    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dce1169-5d8a-46c0-ac53-31fe92c82eae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.426404 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.426245    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68v8\" (UniqueName: \"kubernetes.io/projected/2dce1169-5d8a-46c0-ac53-31fe92c82eae-kube-api-access-p68v8\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.527401 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.527327    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dce1169-5d8a-46c0-ac53-31fe92c82eae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.527401 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.527377    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68v8\" (UniqueName: \"kubernetes.io/projected/2dce1169-5d8a-46c0-ac53-31fe92c82eae-kube-api-access-p68v8\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.529923 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.529902    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dce1169-5d8a-46c0-ac53-31fe92c82eae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.537819 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.537793    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68v8\" (UniqueName: \"kubernetes.io/projected/2dce1169-5d8a-46c0-ac53-31fe92c82eae-kube-api-access-p68v8\") pod \"monitoring-plugin-5f68df7f58-2httj\" (UID: \"2dce1169-5d8a-46c0-ac53-31fe92c82eae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.591587 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.591564    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:54.716575 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.716536    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qslt6" event={"ID":"4867a0c0-f343-4ff2-bdf9-e4dbc9be4761","Type":"ContainerStarted","Data":"b6c03aa99a92a899c8e1b2e72bbd9174a6a54e1f4dc2b4469a3ee5e904409e9f"}
Jul 10 18:59:54.743654 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.743089    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"]
Jul 10 18:59:54.744794 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:54.744748    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qslt6" podStartSLOduration=37.736291878 podStartE2EDuration="44.744713449s" totalImagesPullingTime="7.008421571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:54.743369408 +0000 UTC m=+81.041720778" watchObservedRunningTime="2026-07-10 18:59:54.744713449 +0000 UTC m=+81.043064851"
Jul 10 18:59:54.746563 ip-10-0-139-20 kubenswrapper[2550]: W0710 18:59:54.746535    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dce1169_5d8a_46c0_ac53_31fe92c82eae.slice/crio-452249ec648be738a2bb3a650d20179d5596c39440978834fb3067e03834d8a2 WatchSource:0}: Error finding container 452249ec648be738a2bb3a650d20179d5596c39440978834fb3067e03834d8a2: Status 404 returned error can't find the container with id 452249ec648be738a2bb3a650d20179d5596c39440978834fb3067e03834d8a2
Jul 10 18:59:55.721797 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:55.721764    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj" event={"ID":"2dce1169-5d8a-46c0-ac53-31fe92c82eae","Type":"ContainerStarted","Data":"452249ec648be738a2bb3a650d20179d5596c39440978834fb3067e03834d8a2"}
Jul 10 18:59:56.727252 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.727161    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"78c5e0a14e0d60b89bd01d5a61b14e1bc9110f00f0011e74aba20af0ca4a28a0"}
Jul 10 18:59:56.727252 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.727244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"b4d46681b96840771b572e38a5113d13cfe46ff60f523f68ccbfc6a9195d3982"}
Jul 10 18:59:56.727699 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.727283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"c0b949c95b5fd299be5954a2e672b6d82e9f8e044c4284d9338e2159953daec4"}
Jul 10 18:59:56.728714 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.728688    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj" event={"ID":"2dce1169-5d8a-46c0-ac53-31fe92c82eae","Type":"ContainerStarted","Data":"b29d8f08702f3160766c39b7af368a8919c5a56fe620e129dacb49c431ca1281"}
Jul 10 18:59:56.728940 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.728916    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:56.733572 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.733553    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj"
Jul 10 18:59:56.749343 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:56.749301    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-2httj" podStartSLOduration=1.094194021 podStartE2EDuration="2.749289667s" totalImagesPullingTime="1.655095646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:56.747586832 +0000 UTC m=+83.045938200" watchObservedRunningTime="2026-07-10 18:59:56.749289667 +0000 UTC m=+83.047641036"
Jul 10 18:59:57.734887 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:57.734846    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"687d8c7dd1cf6e9b1f5e56886ebd99dede7da443e813195428efabe7dcba9989"}
Jul 10 18:59:57.735301 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:57.734962    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"3c8ae50a2ac488451fb3eab9cf818e6e156c894824a9d1eef0f52736962dd5c3"}
Jul 10 18:59:57.735301 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:57.735166    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 18:59:57.735301 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:57.735233    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" event={"ID":"5d184c2b-5f86-40f4-8b51-0c94bccf42a5","Type":"ContainerStarted","Data":"393b021ad8328f9e53da7a06d90de176bfdd7a5a673c65dc677ca076c709da01"}
Jul 10 18:59:57.762269 ip-10-0-139-20 kubenswrapper[2550]: I0710 18:59:57.762111    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6" podStartSLOduration=1.9465653120000002 podStartE2EDuration="5.762092542s" totalImagesPullingTime="3.81552723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:57.76002317 +0000 UTC m=+84.058374542" watchObservedRunningTime="2026-07-10 18:59:57.762092542 +0000 UTC m=+84.060443913"
Jul 10 19:00:03.744289 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:00:03.744251    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84f4cb5799-vd4h6"
Jul 10 19:00:15.618146 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:00:15.618118    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ww9lx_50b1ee8c-3513-412b-bce4-68e0dcfe1b79/serve-healthcheck-canary/0.log"
Jul 10 19:00:18.683317 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:00:18.683289    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sfhhk"
Jul 10 19:03:34.096250 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:03:34.096223    2550 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 19:05:32.580715 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.580679    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:05:32.583127 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.583110    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.597779 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.597756    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 10 19:05:32.597885 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.597756    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tcnd9\""
Jul 10 19:05:32.598065 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.598051    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 19:05:32.598174 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.598160    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 19:05:32.618493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.618468    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:05:32.656752 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.656700    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghwf\" (UniqueName: \"kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.656924 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.656821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.676545 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.676512    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-b5vkl"]
Jul 10 19:05:32.678682 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.678667    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.682001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.681977    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gztcg\""
Jul 10 19:05:32.682001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.681980    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:05:32.695459 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.695433    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b5vkl"]
Jul 10 19:05:32.758112 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.758076    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.758266 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.758125    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghwf\" (UniqueName: \"kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.758266 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.758162    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3395d696-33cd-431e-8006-b81f52bb1377-data\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.758266 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.758189    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7t2\" (UniqueName: \"kubernetes.io/projected/3395d696-33cd-431e-8006-b81f52bb1377-kube-api-access-4c7t2\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.760591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.760570    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.768363 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.768340    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghwf\" (UniqueName: \"kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf\") pod \"kserve-controller-manager-69f6979966-4s7g8\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") " pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.859475 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.859377    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3395d696-33cd-431e-8006-b81f52bb1377-data\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.859475 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.859418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7t2\" (UniqueName: \"kubernetes.io/projected/3395d696-33cd-431e-8006-b81f52bb1377-kube-api-access-4c7t2\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.859859 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.859834    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3395d696-33cd-431e-8006-b81f52bb1377-data\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.871988 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.871967    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7t2\" (UniqueName: \"kubernetes.io/projected/3395d696-33cd-431e-8006-b81f52bb1377-kube-api-access-4c7t2\") pod \"seaweedfs-64568bcd49-b5vkl\" (UID: \"3395d696-33cd-431e-8006-b81f52bb1377\") " pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:32.894881 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.894850    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:32.987168 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:32.987132    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:33.049678 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:33.049649    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:05:33.053993 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:05:33.053970    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee689bf_2c1a_4d66_83d6_23785a701ec4.slice/crio-b82a5a856d5dcd5e8dfd3331b5ceec979cacdf8815116b62aa194b4e0d6236d2 WatchSource:0}: Error finding container b82a5a856d5dcd5e8dfd3331b5ceec979cacdf8815116b62aa194b4e0d6236d2: Status 404 returned error can't find the container with id b82a5a856d5dcd5e8dfd3331b5ceec979cacdf8815116b62aa194b4e0d6236d2
Jul 10 19:05:33.055234 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:33.055218    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:05:33.138652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:33.138577    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b5vkl"]
Jul 10 19:05:33.141993 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:05:33.141962    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3395d696_33cd_431e_8006_b81f52bb1377.slice/crio-4d140c18e9074a2e1cf8493e5d0e4cf8f189fbc25ce00404bb3314b9ae80ca33 WatchSource:0}: Error finding container 4d140c18e9074a2e1cf8493e5d0e4cf8f189fbc25ce00404bb3314b9ae80ca33: Status 404 returned error can't find the container with id 4d140c18e9074a2e1cf8493e5d0e4cf8f189fbc25ce00404bb3314b9ae80ca33
Jul 10 19:05:33.884156 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:33.884119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b5vkl" event={"ID":"3395d696-33cd-431e-8006-b81f52bb1377","Type":"ContainerStarted","Data":"4d140c18e9074a2e1cf8493e5d0e4cf8f189fbc25ce00404bb3314b9ae80ca33"}
Jul 10 19:05:33.886566 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:33.886532    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" event={"ID":"1ee689bf-2c1a-4d66-83d6-23785a701ec4","Type":"ContainerStarted","Data":"b82a5a856d5dcd5e8dfd3331b5ceec979cacdf8815116b62aa194b4e0d6236d2"}
Jul 10 19:05:36.638563 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.638537    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:05:36.902186 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.902139    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" event={"ID":"1ee689bf-2c1a-4d66-83d6-23785a701ec4","Type":"ContainerStarted","Data":"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"}
Jul 10 19:05:36.902380 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.902295    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:05:36.903690 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.903665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b5vkl" event={"ID":"3395d696-33cd-431e-8006-b81f52bb1377","Type":"ContainerStarted","Data":"1ddf8d0cae605315fb11541f9a07f356fe8f9589a3e766f1d33816c4049b9b76"}
Jul 10 19:05:36.903795 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.903713    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:05:36.924171 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.924094    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" podStartSLOduration=1.411728847 podStartE2EDuration="4.924079826s" totalImagesPullingTime="3.512350979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:36.92293912 +0000 UTC m=+423.221290490" watchObservedRunningTime="2026-07-10 19:05:36.924079826 +0000 UTC m=+423.222431195"
Jul 10 19:05:36.942310 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:36.942264    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-b5vkl" podStartSLOduration=1.450488926 podStartE2EDuration="4.942250932s" totalImagesPullingTime="3.491762006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:36.940458928 +0000 UTC m=+423.238810300" watchObservedRunningTime="2026-07-10 19:05:36.942250932 +0000 UTC m=+423.240602301"
Jul 10 19:05:42.909499 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:05:42.909468    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-b5vkl"
Jul 10 19:06:07.911977 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:07.911896    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:06:08.326487 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.326400    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:06:08.326635 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.326568    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" podUID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" containerName="manager" containerID="cri-o://ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4" gracePeriod=10
Jul 10 19:06:08.567143 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.567118    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:06:08.732974 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.732921    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert\") pod \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") "
Jul 10 19:06:08.732974 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.732986    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xghwf\" (UniqueName: \"kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf\") pod \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\" (UID: \"1ee689bf-2c1a-4d66-83d6-23785a701ec4\") "
Jul 10 19:06:08.735285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.735239    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert" (OuterVolumeSpecName: "cert") pod "1ee689bf-2c1a-4d66-83d6-23785a701ec4" (UID: "1ee689bf-2c1a-4d66-83d6-23785a701ec4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:06:08.735285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.735275    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf" (OuterVolumeSpecName: "kube-api-access-xghwf") pod "1ee689bf-2c1a-4d66-83d6-23785a701ec4" (UID: "1ee689bf-2c1a-4d66-83d6-23785a701ec4"). InnerVolumeSpecName "kube-api-access-xghwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:06:08.833823 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.833790    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xghwf\" (UniqueName: \"kubernetes.io/projected/1ee689bf-2c1a-4d66-83d6-23785a701ec4-kube-api-access-xghwf\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:06:08.833823 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:08.833819    2550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ee689bf-2c1a-4d66-83d6-23785a701ec4-cert\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:06:09.022980 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.022891    2550 generic.go:356] "Generic (PLEG): container finished" podID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" containerID="ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4" exitCode=0
Jul 10 19:06:09.022980 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.022940    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69f6979966-4s7g8"
Jul 10 19:06:09.023493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.022985    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" event={"ID":"1ee689bf-2c1a-4d66-83d6-23785a701ec4","Type":"ContainerDied","Data":"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"}
Jul 10 19:06:09.023493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.023076    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69f6979966-4s7g8" event={"ID":"1ee689bf-2c1a-4d66-83d6-23785a701ec4","Type":"ContainerDied","Data":"b82a5a856d5dcd5e8dfd3331b5ceec979cacdf8815116b62aa194b4e0d6236d2"}
Jul 10 19:06:09.023493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.023104    2550 scope.go:117] "RemoveContainer" containerID="ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"
Jul 10 19:06:09.032130 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.032106    2550 scope.go:117] "RemoveContainer" containerID="ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"
Jul 10 19:06:09.032473 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:06:09.032441    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4\": container with ID starting with ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4 not found: ID does not exist" containerID="ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"
Jul 10 19:06:09.032542 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.032480    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4"} err="failed to get container status \"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4\": rpc error: code = NotFound desc = could not find container \"ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4\": container with ID starting with ff1a828152cc23893d9a77bb7b2e174f1f18e007fe15d342cba8de2a8f7f7df4 not found: ID does not exist"
Jul 10 19:06:09.044139 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.044107    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:06:09.049478 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:09.049451    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-69f6979966-4s7g8"]
Jul 10 19:06:10.274600 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:10.274560    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" path="/var/lib/kubelet/pods/1ee689bf-2c1a-4d66-83d6-23785a701ec4/volumes"
Jul 10 19:06:44.318528 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.318490    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wr4nb"]
Jul 10 19:06:44.319036 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.318777    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" containerName="manager"
Jul 10 19:06:44.319036 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.318788    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" containerName="manager"
Jul 10 19:06:44.319036 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.318834    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee689bf-2c1a-4d66-83d6-23785a701ec4" containerName="manager"
Jul 10 19:06:44.320689 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.320671    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.323427 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.323407    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 10 19:06:44.324650 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.324629    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zfjbx\""
Jul 10 19:06:44.329341 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.329315    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-rhf8n"]
Jul 10 19:06:44.331303 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.331287    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.338003 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.337984    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 10 19:06:44.338169 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.338148    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-26cn6\""
Jul 10 19:06:44.339426 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.339403    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wr4nb"]
Jul 10 19:06:44.358911 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.358875    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-rhf8n"]
Jul 10 19:06:44.389673 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.389631    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8v9\" (UniqueName: \"kubernetes.io/projected/6306cc0c-bfb1-4336-81b8-01d6e34773c7-kube-api-access-fs8v9\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.389673 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.389674    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6306cc0c-bfb1-4336-81b8-01d6e34773c7-tls-certs\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.389915 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.389776    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f38207-8fdf-441b-95a4-ac367975d26f-cert\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.389915 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.389834    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlq7\" (UniqueName: \"kubernetes.io/projected/96f38207-8fdf-441b-95a4-ac367975d26f-kube-api-access-4wlq7\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.490978 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.490938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlq7\" (UniqueName: \"kubernetes.io/projected/96f38207-8fdf-441b-95a4-ac367975d26f-kube-api-access-4wlq7\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.490978 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.490978    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8v9\" (UniqueName: \"kubernetes.io/projected/6306cc0c-bfb1-4336-81b8-01d6e34773c7-kube-api-access-fs8v9\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.491246 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.490999    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6306cc0c-bfb1-4336-81b8-01d6e34773c7-tls-certs\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.491246 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.491032    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f38207-8fdf-441b-95a4-ac367975d26f-cert\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.493592 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.493565    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6306cc0c-bfb1-4336-81b8-01d6e34773c7-tls-certs\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.493846 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.493822    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96f38207-8fdf-441b-95a4-ac367975d26f-cert\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.501250 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.501221    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8v9\" (UniqueName: \"kubernetes.io/projected/6306cc0c-bfb1-4336-81b8-01d6e34773c7-kube-api-access-fs8v9\") pod \"model-serving-api-fd65d7d6-rhf8n\" (UID: \"6306cc0c-bfb1-4336-81b8-01d6e34773c7\") " pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.501469 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.501451    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlq7\" (UniqueName: \"kubernetes.io/projected/96f38207-8fdf-441b-95a4-ac367975d26f-kube-api-access-4wlq7\") pod \"odh-model-controller-84b54b7f97-wr4nb\" (UID: \"96f38207-8fdf-441b-95a4-ac367975d26f\") " pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.632059 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.631970    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:44.640254 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.640232    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:44.783485 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.783453    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wr4nb"]
Jul 10 19:06:44.787425 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:06:44.787403    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f38207_8fdf_441b_95a4_ac367975d26f.slice/crio-c49ab0164d9f6a9f51500160802ce21266c01bb8e1a76dc078a680f2174ab177 WatchSource:0}: Error finding container c49ab0164d9f6a9f51500160802ce21266c01bb8e1a76dc078a680f2174ab177: Status 404 returned error can't find the container with id c49ab0164d9f6a9f51500160802ce21266c01bb8e1a76dc078a680f2174ab177
Jul 10 19:06:44.799594 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:44.799574    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-rhf8n"]
Jul 10 19:06:44.801700 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:06:44.801669    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6306cc0c_bfb1_4336_81b8_01d6e34773c7.slice/crio-192a9b86b610631bd69559e938d593907f0c5433f5d6d0dfd56623dc2da76d43 WatchSource:0}: Error finding container 192a9b86b610631bd69559e938d593907f0c5433f5d6d0dfd56623dc2da76d43: Status 404 returned error can't find the container with id 192a9b86b610631bd69559e938d593907f0c5433f5d6d0dfd56623dc2da76d43
Jul 10 19:06:45.148669 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:45.148625    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-rhf8n" event={"ID":"6306cc0c-bfb1-4336-81b8-01d6e34773c7","Type":"ContainerStarted","Data":"192a9b86b610631bd69559e938d593907f0c5433f5d6d0dfd56623dc2da76d43"}
Jul 10 19:06:45.149932 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:45.149902    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-wr4nb" event={"ID":"96f38207-8fdf-441b-95a4-ac367975d26f","Type":"ContainerStarted","Data":"c49ab0164d9f6a9f51500160802ce21266c01bb8e1a76dc078a680f2174ab177"}
Jul 10 19:06:46.157044 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:46.157005    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-rhf8n" event={"ID":"6306cc0c-bfb1-4336-81b8-01d6e34773c7","Type":"ContainerStarted","Data":"0873ad6501edd885063ef7acf760338ab67e8f759ac8e3dcebf75940d40d8d62"}
Jul 10 19:06:46.157525 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:46.157115    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:46.177781 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:46.177398    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-rhf8n" podStartSLOduration=0.959917112 podStartE2EDuration="2.177375993s" totalImagesPullingTime="1.217458881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:46.174883967 +0000 UTC m=+492.473235337" watchObservedRunningTime="2026-07-10 19:06:46.177375993 +0000 UTC m=+492.475727362"
Jul 10 19:06:48.166334 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:48.166300    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-wr4nb" event={"ID":"96f38207-8fdf-441b-95a4-ac367975d26f","Type":"ContainerStarted","Data":"cd8a03f3effea6b04c5f0279e01e44f52f1f1d1a7e64c07aba9297f739bf5719"}
Jul 10 19:06:48.166780 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:48.166415    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:06:48.184315 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:48.184172    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-wr4nb" podStartSLOduration=1.54795905 podStartE2EDuration="4.184151098s" totalImagesPullingTime="2.636192048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:48.183638325 +0000 UTC m=+494.481989715" watchObservedRunningTime="2026-07-10 19:06:48.184151098 +0000 UTC m=+494.482502468"
Jul 10 19:06:57.165537 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:57.165509    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-rhf8n"
Jul 10 19:06:59.173494 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:06:59.173459    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-wr4nb"
Jul 10 19:08:52.222028 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.221990    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:08:52.225563 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.225539    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.228006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.227981    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:08:52.228914 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.228831    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ac00e-predictor-serving-cert\""
Jul 10 19:08:52.228914 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.228864    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:08:52.228914 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.228877    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hm6ps\""
Jul 10 19:08:52.229158 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.228864    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ac00e-kube-rbac-proxy-sar-config\""
Jul 10 19:08:52.233293 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.232895    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:08:52.315260 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.315230    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.315421 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.315287    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.315484 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.315407    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx95j\" (UniqueName: \"kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.416340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.416303    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.416520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.416371    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx95j\" (UniqueName: \"kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.416520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.416412    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.416520 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:08:52.416466    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-serving-cert: secret "success-200-isvc-ac00e-predictor-serving-cert" not found
Jul 10 19:08:52.416677 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:08:52.416557    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls podName:5a961d5d-c8a2-483f-82c4-d63f496e434f nodeName:}" failed. No retries permitted until 2026-07-10 19:08:52.916527184 +0000 UTC m=+619.214878549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls") pod "success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" (UID: "5a961d5d-c8a2-483f-82c4-d63f496e434f") : secret "success-200-isvc-ac00e-predictor-serving-cert" not found
Jul 10 19:08:52.417086 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.417066    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.422098 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.422069    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:08:52.425954 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.425935    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.425954 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.425944    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx95j\" (UniqueName: \"kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.428094 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.428075    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ac00e-predictor-serving-cert\""
Jul 10 19:08:52.428235 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.428219    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ac00e-kube-rbac-proxy-sar-config\""
Jul 10 19:08:52.435077 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.435056    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:08:52.517681 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.517578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mgh\" (UniqueName: \"kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.517681 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.517632    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.517904 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.517695    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.529472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.529445    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:08:52.532871 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.532847    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.535453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.535256    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:08:52.535453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.535275    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:08:52.535453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.535323    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 10 19:08:52.535453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.535255    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 10 19:08:52.543520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.543347    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:08:52.618271 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618242    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.618436 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618304    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.618436 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618376    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mgh\" (UniqueName: \"kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.618436 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618424    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.618588 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618469    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qk76\" (UniqueName: \"kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.618588 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618494    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.618588 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618526    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.618588 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.618551    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.619123 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.619106    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.621096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.621076    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.628099 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.628076    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mgh\" (UniqueName: \"kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh\") pod \"error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") " pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.719376 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.719333    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qk76\" (UniqueName: \"kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.719376 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.719372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.719376 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.719395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.719693 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.719466    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.719693 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.719495    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.720167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.720129    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.720287 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.720163    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.720287 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.720259    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.722436 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.722415    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.730255 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.730232    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qk76\" (UniqueName: \"kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76\") pod \"isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.743887 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.743868    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:08:52.844632 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.844335    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:08:52.881918 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.881892    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:08:52.883751 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:08:52.883710    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030651ea_b258_46f1_be66_6c973c0b82fe.slice/crio-acc7b6b21734313cd756a1f3c1c8c765bf803a12e7b13912587afbb891c1b64e WatchSource:0}: Error finding container acc7b6b21734313cd756a1f3c1c8c765bf803a12e7b13912587afbb891c1b64e: Status 404 returned error can't find the container with id acc7b6b21734313cd756a1f3c1c8c765bf803a12e7b13912587afbb891c1b64e
Jul 10 19:08:52.921407 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.921381    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:52.921560 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:08:52.921518    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-serving-cert: secret "success-200-isvc-ac00e-predictor-serving-cert" not found
Jul 10 19:08:52.921623 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:08:52.921578    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls podName:5a961d5d-c8a2-483f-82c4-d63f496e434f nodeName:}" failed. No retries permitted until 2026-07-10 19:08:53.92156191 +0000 UTC m=+620.219913258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls") pod "success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" (UID: "5a961d5d-c8a2-483f-82c4-d63f496e434f") : secret "success-200-isvc-ac00e-predictor-serving-cert" not found
Jul 10 19:08:52.981375 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:52.981233    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:08:52.984015 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:08:52.983990    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fdc5e7_4a67_4ee2_8b3f_3fa819d62c95.slice/crio-3f9fdf58e6781e7a8a159af66158843430491e790295c1d2f53a10e9ac3222ca WatchSource:0}: Error finding container 3f9fdf58e6781e7a8a159af66158843430491e790295c1d2f53a10e9ac3222ca: Status 404 returned error can't find the container with id 3f9fdf58e6781e7a8a159af66158843430491e790295c1d2f53a10e9ac3222ca
Jul 10 19:08:53.611751 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:53.611594    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerStarted","Data":"3f9fdf58e6781e7a8a159af66158843430491e790295c1d2f53a10e9ac3222ca"}
Jul 10 19:08:53.613628 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:53.613559    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerStarted","Data":"acc7b6b21734313cd756a1f3c1c8c765bf803a12e7b13912587afbb891c1b64e"}
Jul 10 19:08:53.931931 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:53.931345    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:53.946938 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:53.946870    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") " pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:54.038424 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:54.038102    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:08:54.241421 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:54.241049    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:08:54.257818 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:08:54.257771    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a961d5d_c8a2_483f_82c4_d63f496e434f.slice/crio-c45d192e71aa5335b93e0f2b554333fe88043facdeb8e4475aa7ede792fec0c4 WatchSource:0}: Error finding container c45d192e71aa5335b93e0f2b554333fe88043facdeb8e4475aa7ede792fec0c4: Status 404 returned error can't find the container with id c45d192e71aa5335b93e0f2b554333fe88043facdeb8e4475aa7ede792fec0c4
Jul 10 19:08:54.623120 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:08:54.622950    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerStarted","Data":"c45d192e71aa5335b93e0f2b554333fe88043facdeb8e4475aa7ede792fec0c4"}
Jul 10 19:09:10.722327 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:10.722287    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerStarted","Data":"884a96c4bcef9d4a23e92e6429ca9676877502c00c0678fab263900a2aa0de31"}
Jul 10 19:09:10.723989 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:10.723945    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerStarted","Data":"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"}
Jul 10 19:09:10.725457 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:10.725434    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerStarted","Data":"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"}
Jul 10 19:09:13.740173 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.740128    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerStarted","Data":"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"}
Jul 10 19:09:13.740995 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.740209    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:09:13.740995 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.740352    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:09:13.741768 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.741716    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:13.742232 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.742209    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerStarted","Data":"7a73f7915c82753e13dfc97b68f015f255dc67ae5012a44d653c0ad48d050d9c"}
Jul 10 19:09:13.742496 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.742478    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:09:13.742575 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.742529    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:09:13.743487 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.743463    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:13.757868 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.757829    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podStartSLOduration=1.63779919 podStartE2EDuration="21.757815224s" totalImagesPullingTime="20.120016034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:13.756605336 +0000 UTC m=+640.054956707" watchObservedRunningTime="2026-07-10 19:09:13.757815224 +0000 UTC m=+640.056166594"
Jul 10 19:09:13.774025 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:13.773983    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podStartSLOduration=3.040118109 podStartE2EDuration="21.773974195s" totalImagesPullingTime="18.733856086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:13.772792468 +0000 UTC m=+640.071143837" watchObservedRunningTime="2026-07-10 19:09:13.773974195 +0000 UTC m=+640.072325563"
Jul 10 19:09:14.747900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:14.747867    2550 generic.go:356] "Generic (PLEG): container finished" podID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerID="1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156" exitCode=0
Jul 10 19:09:14.748295 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:14.747943    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerDied","Data":"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"}
Jul 10 19:09:14.748395 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:14.748366    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:14.748547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:14.748509    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:19.753663 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:19.753625    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:09:19.822001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:19.754412    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:19.822001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:19.754472    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:09:19.822001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:19.754928    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:22.782692 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:22.782664    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerStarted","Data":"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"}
Jul 10 19:09:23.788070 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:23.788032    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerStarted","Data":"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"}
Jul 10 19:09:23.788655 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:23.788146    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:09:23.808972 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:23.808909    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podStartSLOduration=3.045710245 podStartE2EDuration="31.80888851s" totalImagesPullingTime="25.215465s" totalInitContainerRuntime="3.547713265s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:23.80732239 +0000 UTC m=+650.105673766" watchObservedRunningTime="2026-07-10 19:09:23.80888851 +0000 UTC m=+650.107239879"
Jul 10 19:09:24.792312 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:24.792281    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:09:24.793438 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:24.793409    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:09:25.795895 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:25.795853    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:09:29.755152 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:29.755116    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:29.755509 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:29.755116    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:30.800160 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:30.800130    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:09:30.800787 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:30.800758    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:09:39.755446 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:39.755407    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:39.755900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:39.755407    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:40.801030 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:40.800990    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:09:49.754541 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:49.754505    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:09:49.755005 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:49.754962    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 10 19:09:50.801608 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:50.801569    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:09:59.755372 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:59.755343    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:09:59.755806 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:09:59.755420    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:10:00.800690 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:00.800649    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:10:10.800919 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:10.800873    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:10:20.801596 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:20.801555    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:10:26.236813 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.236700    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:10:26.237186 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.237005    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" containerID="cri-o://884a96c4bcef9d4a23e92e6429ca9676877502c00c0678fab263900a2aa0de31" gracePeriod=30
Jul 10 19:10:26.237186 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.237048    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kube-rbac-proxy" containerID="cri-o://7a73f7915c82753e13dfc97b68f015f255dc67ae5012a44d653c0ad48d050d9c" gracePeriod=30
Jul 10 19:10:26.294906 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.294875    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:10:26.295120 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.295099    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container" containerID="cri-o://c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe" gracePeriod=30
Jul 10 19:10:26.295251 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.295217    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kube-rbac-proxy" containerID="cri-o://b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d" gracePeriod=30
Jul 10 19:10:26.359121 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.359090    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:10:26.362289 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.362263    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.364932 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.364910    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4aae5-predictor-serving-cert\""
Jul 10 19:10:26.365119 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.364929    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4aae5-kube-rbac-proxy-sar-config\""
Jul 10 19:10:26.375300 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.375274    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:10:26.419140 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.419117    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:10:26.422301 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.422286    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.424816 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.424796    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4aae5-predictor-serving-cert\""
Jul 10 19:10:26.424816 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.424810    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4aae5-kube-rbac-proxy-sar-config\""
Jul 10 19:10:26.432404 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.432385    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:10:26.490514 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.490446    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.490629 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.490529    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644zp\" (UniqueName: \"kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.490629 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.490575    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.591333 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591298    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqx9\" (UniqueName: \"kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.591333 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.591552 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591362    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.591552 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591423    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.591552 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591513    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644zp\" (UniqueName: \"kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.591699 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.591561    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.592027 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.592009    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.594168 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.594143    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.600942 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.600922    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-644zp\" (UniqueName: \"kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp\") pod \"success-200-isvc-4aae5-predictor-f8cd6867f-plzsm\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") " pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.675884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.675851    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:26.692768 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.692351    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.692768 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.692446    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.692768 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.692512    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqx9\" (UniqueName: \"kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.693158 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:10:26.693130    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-serving-cert: secret "error-404-isvc-4aae5-predictor-serving-cert" not found
Jul 10 19:10:26.693245 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.693198    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.693245 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:10:26.693207    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls podName:69e78f05-d420-4352-bb49-9336e87fa5e8 nodeName:}" failed. No retries permitted until 2026-07-10 19:10:27.193187231 +0000 UTC m=+713.491538583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls") pod "error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" (UID: "69e78f05-d420-4352-bb49-9336e87fa5e8") : secret "error-404-isvc-4aae5-predictor-serving-cert" not found
Jul 10 19:10:26.701441 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.701418    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqx9\" (UniqueName: \"kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:26.812849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:26.812684    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:10:26.815049 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:10:26.815021    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68099bf1_5bb6_470a_b8f1_5e3a611f0679.slice/crio-bcc380f7b9a4189e7299a3375a63319834560da69a19951202a10bedb47dd14a WatchSource:0}: Error finding container bcc380f7b9a4189e7299a3375a63319834560da69a19951202a10bedb47dd14a: Status 404 returned error can't find the container with id bcc380f7b9a4189e7299a3375a63319834560da69a19951202a10bedb47dd14a
Jul 10 19:10:27.033948 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.033859    2550 generic.go:356] "Generic (PLEG): container finished" podID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerID="7a73f7915c82753e13dfc97b68f015f255dc67ae5012a44d653c0ad48d050d9c" exitCode=2
Jul 10 19:10:27.033948 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.033934    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerDied","Data":"7a73f7915c82753e13dfc97b68f015f255dc67ae5012a44d653c0ad48d050d9c"}
Jul 10 19:10:27.035872 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.035843    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerStarted","Data":"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"}
Jul 10 19:10:27.036016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.035891    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerStarted","Data":"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"}
Jul 10 19:10:27.036016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.035935    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:27.036159 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.036064    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:27.036159 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.036101    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerStarted","Data":"bcc380f7b9a4189e7299a3375a63319834560da69a19951202a10bedb47dd14a"}
Jul 10 19:10:27.037572 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.037547    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:10:27.038254 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.038233    2550 generic.go:356] "Generic (PLEG): container finished" podID="030651ea-b258-46f1-be66-6c973c0b82fe" containerID="b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d" exitCode=2
Jul 10 19:10:27.038331 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.038274    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerDied","Data":"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"}
Jul 10 19:10:27.055822 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.055769    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podStartSLOduration=1.055721557 podStartE2EDuration="1.055721557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:27.054842394 +0000 UTC m=+713.353193765" watchObservedRunningTime="2026-07-10 19:10:27.055721557 +0000 UTC m=+713.354072926"
Jul 10 19:10:27.197154 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.197119    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:27.199649 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.199622    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") pod \"error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") " pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:27.332884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.332790    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:27.464887 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:27.464862    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:10:27.467178 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:10:27.467152    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e78f05_d420_4352_bb49_9336e87fa5e8.slice/crio-5b10a3084ee5403f8be28885198b3917647015fc1c3ce2ef737e7826e716c520 WatchSource:0}: Error finding container 5b10a3084ee5403f8be28885198b3917647015fc1c3ce2ef737e7826e716c520: Status 404 returned error can't find the container with id 5b10a3084ee5403f8be28885198b3917647015fc1c3ce2ef737e7826e716c520
Jul 10 19:10:28.043721 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:28.043685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerStarted","Data":"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"}
Jul 10 19:10:28.043904 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:28.043769    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerStarted","Data":"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"}
Jul 10 19:10:28.043904 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:28.043791    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerStarted","Data":"5b10a3084ee5403f8be28885198b3917647015fc1c3ce2ef737e7826e716c520"}
Jul 10 19:10:28.044042 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:28.044015    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:10:28.062153 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:28.062059    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podStartSLOduration=2.062039448 podStartE2EDuration="2.062039448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:28.061345852 +0000 UTC m=+714.359697223" watchObservedRunningTime="2026-07-10 19:10:28.062039448 +0000 UTC m=+714.360390820"
Jul 10 19:10:29.048271 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.048235    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:29.048687 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.048302    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:29.049532 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.049504    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:10:29.647594 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.647572    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:10:29.748902 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.748792    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 10 19:10:29.755190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.755161    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 10 19:10:29.823161 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.823129    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64mgh\" (UniqueName: \"kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh\") pod \"030651ea-b258-46f1-be66-6c973c0b82fe\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") "
Jul 10 19:10:29.823321 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.823190    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls\") pod \"030651ea-b258-46f1-be66-6c973c0b82fe\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") "
Jul 10 19:10:29.823321 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.823209    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config\") pod \"030651ea-b258-46f1-be66-6c973c0b82fe\" (UID: \"030651ea-b258-46f1-be66-6c973c0b82fe\") "
Jul 10 19:10:29.823621 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.823598    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "030651ea-b258-46f1-be66-6c973c0b82fe" (UID: "030651ea-b258-46f1-be66-6c973c0b82fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:29.825540 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.825511    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh" (OuterVolumeSpecName: "kube-api-access-64mgh") pod "030651ea-b258-46f1-be66-6c973c0b82fe" (UID: "030651ea-b258-46f1-be66-6c973c0b82fe"). InnerVolumeSpecName "kube-api-access-64mgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:29.825605 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.825537    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "030651ea-b258-46f1-be66-6c973c0b82fe" (UID: "030651ea-b258-46f1-be66-6c973c0b82fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:29.924493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.924460    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/030651ea-b258-46f1-be66-6c973c0b82fe-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:29.924493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.924489    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/030651ea-b258-46f1-be66-6c973c0b82fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:29.924493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:29.924508    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mgh\" (UniqueName: \"kubernetes.io/projected/030651ea-b258-46f1-be66-6c973c0b82fe-kube-api-access-64mgh\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:30.054348 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.054318    2550 generic.go:356] "Generic (PLEG): container finished" podID="030651ea-b258-46f1-be66-6c973c0b82fe" containerID="c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe" exitCode=0
Jul 10 19:10:30.054726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.054370    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"
Jul 10 19:10:30.054726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.054398    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerDied","Data":"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"}
Jul 10 19:10:30.054726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.054462    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm" event={"ID":"030651ea-b258-46f1-be66-6c973c0b82fe","Type":"ContainerDied","Data":"acc7b6b21734313cd756a1f3c1c8c765bf803a12e7b13912587afbb891c1b64e"}
Jul 10 19:10:30.054726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.054501    2550 scope.go:117] "RemoveContainer" containerID="b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"
Jul 10 19:10:30.056545 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.056509    2550 generic.go:356] "Generic (PLEG): container finished" podID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerID="884a96c4bcef9d4a23e92e6429ca9676877502c00c0678fab263900a2aa0de31" exitCode=0
Jul 10 19:10:30.056635 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.056560    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerDied","Data":"884a96c4bcef9d4a23e92e6429ca9676877502c00c0678fab263900a2aa0de31"}
Jul 10 19:10:30.057094 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.057063    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:10:30.067951 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.067929    2550 scope.go:117] "RemoveContainer" containerID="c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"
Jul 10 19:10:30.076465 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.076444    2550 scope.go:117] "RemoveContainer" containerID="b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"
Jul 10 19:10:30.076833 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:10:30.076797    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d\": container with ID starting with b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d not found: ID does not exist" containerID="b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"
Jul 10 19:10:30.076935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.076843    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d"} err="failed to get container status \"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d\": rpc error: code = NotFound desc = could not find container \"b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d\": container with ID starting with b0d498f2906c0c349dba6847439f597cbfac96539c23dff3cca41005105b3d3d not found: ID does not exist"
Jul 10 19:10:30.076935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.076867    2550 scope.go:117] "RemoveContainer" containerID="c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"
Jul 10 19:10:30.076935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.076892    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:10:30.077153 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:10:30.077130    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe\": container with ID starting with c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe not found: ID does not exist" containerID="c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"
Jul 10 19:10:30.077196 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.077166    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe"} err="failed to get container status \"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe\": rpc error: code = NotFound desc = could not find container \"c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe\": container with ID starting with c3be3e087d06832a51175e8e1d7d344200e475f7b8edf7269320af2fa48310fe not found: ID does not exist"
Jul 10 19:10:30.079643 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.079620    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ac00e-predictor-6b6fb6bc7b-dm4wm"]
Jul 10 19:10:30.185054 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.185025    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:10:30.226580 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.226545    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx95j\" (UniqueName: \"kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j\") pod \"5a961d5d-c8a2-483f-82c4-d63f496e434f\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") "
Jul 10 19:10:30.226766 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.226619    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") pod \"5a961d5d-c8a2-483f-82c4-d63f496e434f\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") "
Jul 10 19:10:30.226766 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.226644    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config\") pod \"5a961d5d-c8a2-483f-82c4-d63f496e434f\" (UID: \"5a961d5d-c8a2-483f-82c4-d63f496e434f\") "
Jul 10 19:10:30.227056 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.227033    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a961d5d-c8a2-483f-82c4-d63f496e434f" (UID: "5a961d5d-c8a2-483f-82c4-d63f496e434f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:30.228873 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.228846    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a961d5d-c8a2-483f-82c4-d63f496e434f" (UID: "5a961d5d-c8a2-483f-82c4-d63f496e434f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:30.228953 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.228891    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j" (OuterVolumeSpecName: "kube-api-access-zx95j") pod "5a961d5d-c8a2-483f-82c4-d63f496e434f" (UID: "5a961d5d-c8a2-483f-82c4-d63f496e434f"). InnerVolumeSpecName "kube-api-access-zx95j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:30.275256 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.275181    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" path="/var/lib/kubelet/pods/030651ea-b258-46f1-be66-6c973c0b82fe/volumes"
Jul 10 19:10:30.327598 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.327562    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a961d5d-c8a2-483f-82c4-d63f496e434f-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:30.327598 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.327588    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:30.327598 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.327600    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx95j\" (UniqueName: \"kubernetes.io/projected/5a961d5d-c8a2-483f-82c4-d63f496e434f-kube-api-access-zx95j\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:10:30.800681 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:30.800631    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 10 19:10:31.061918 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.061836    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"
Jul 10 19:10:31.061918 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.061874    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg" event={"ID":"5a961d5d-c8a2-483f-82c4-d63f496e434f","Type":"ContainerDied","Data":"c45d192e71aa5335b93e0f2b554333fe88043facdeb8e4475aa7ede792fec0c4"}
Jul 10 19:10:31.062381 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.061943    2550 scope.go:117] "RemoveContainer" containerID="7a73f7915c82753e13dfc97b68f015f255dc67ae5012a44d653c0ad48d050d9c"
Jul 10 19:10:31.069879 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.069852    2550 scope.go:117] "RemoveContainer" containerID="884a96c4bcef9d4a23e92e6429ca9676877502c00c0678fab263900a2aa0de31"
Jul 10 19:10:31.080223 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.080194    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:10:31.085944 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:31.085923    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ac00e-predictor-5b8ff4ff6f-qrgbg"]
Jul 10 19:10:32.274946 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:32.274913    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" path="/var/lib/kubelet/pods/5a961d5d-c8a2-483f-82c4-d63f496e434f/volumes"
Jul 10 19:10:33.048472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:33.048448    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:10:33.048933 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:33.048909    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:10:35.060637 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:35.060608    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:10:35.061184 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:35.061159    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:10:40.801662 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:40.801635    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:10:43.049248 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:43.049207    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:10:45.061158 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:45.061117    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:10:53.049713 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:53.049677    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:10:55.061506 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:10:55.061468    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:11:02.198275 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.198245    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:11:02.198652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.198486    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container" containerID="cri-o://ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40" gracePeriod=30
Jul 10 19:11:02.198652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.198586    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kube-rbac-proxy" containerID="cri-o://65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb" gracePeriod=30
Jul 10 19:11:02.310569 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.310542    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:11:02.311022 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311005    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container"
Jul 10 19:11:02.311070 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311026    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container"
Jul 10 19:11:02.311070 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311048    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kube-rbac-proxy"
Jul 10 19:11:02.311070 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311058    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kube-rbac-proxy"
Jul 10 19:11:02.311167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311079    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kube-rbac-proxy"
Jul 10 19:11:02.311167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311088    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kube-rbac-proxy"
Jul 10 19:11:02.311167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311099    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container"
Jul 10 19:11:02.311167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311108    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container"
Jul 10 19:11:02.311281 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311181    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kserve-container"
Jul 10 19:11:02.311281 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311192    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a961d5d-c8a2-483f-82c4-d63f496e434f" containerName="kube-rbac-proxy"
Jul 10 19:11:02.311281 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311204    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kserve-container"
Jul 10 19:11:02.311281 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.311215    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="030651ea-b258-46f1-be66-6c973c0b82fe" containerName="kube-rbac-proxy"
Jul 10 19:11:02.314321 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.314302    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.317650 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.317631    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31f26-kube-rbac-proxy-sar-config\""
Jul 10 19:11:02.317879 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.317865    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31f26-predictor-serving-cert\""
Jul 10 19:11:02.329826 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.329800    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:11:02.365659 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.365633    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.365806 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.365676    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf58\" (UniqueName: \"kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.365806 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.365705    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.441129 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.441096    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:11:02.444444 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.444429    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.447091 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.447061    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31f26-kube-rbac-proxy-sar-config\""
Jul 10 19:11:02.447207 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.447143    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31f26-predictor-serving-cert\""
Jul 10 19:11:02.458512 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.458486    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:11:02.467048 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.467022    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.467136 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.467087    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.467186 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.467132    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf58\" (UniqueName: \"kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.467909 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.467881    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.469620 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.469600    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.478318 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.478295    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf58\" (UniqueName: \"kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58\") pod \"success-200-isvc-31f26-predictor-7599c98b7f-ctz52\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") " pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.567768 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.567727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d4r\" (UniqueName: \"kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.567935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.567779    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.567935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.567843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.624162 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.624129    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:02.669239 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.669137    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d4r\" (UniqueName: \"kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.669239 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.669186    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.669239 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.669227    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.670190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.670147    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.672958 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.672529    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.678704 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.678678    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d4r\" (UniqueName: \"kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r\") pod \"error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") " pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.754907 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.754869    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:02.759266 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.759246    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:11:02.761643 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:11:02.761615    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6e9d06_c511_4bfe_9373_701578db33e5.slice/crio-db37d09360f14214e5dbcb0e99880b6116fbb7a9345949b83775c0f0977f3a60 WatchSource:0}: Error finding container db37d09360f14214e5dbcb0e99880b6116fbb7a9345949b83775c0f0977f3a60: Status 404 returned error can't find the container with id db37d09360f14214e5dbcb0e99880b6116fbb7a9345949b83775c0f0977f3a60
Jul 10 19:11:02.899853 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:02.899826    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:11:02.903785 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:11:02.903758    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9cfa8c_7f6a_4a51_80a8_5fe6fee09f23.slice/crio-28ccba479235c17f548e02832e6a225093d9aff2144e6bc73e746d302f88d3d8 WatchSource:0}: Error finding container 28ccba479235c17f548e02832e6a225093d9aff2144e6bc73e746d302f88d3d8: Status 404 returned error can't find the container with id 28ccba479235c17f548e02832e6a225093d9aff2144e6bc73e746d302f88d3d8
Jul 10 19:11:03.049179 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.049146    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:11:03.186931 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.186900    2550 generic.go:356] "Generic (PLEG): container finished" podID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerID="65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb" exitCode=2
Jul 10 19:11:03.187078 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.186973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerDied","Data":"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"}
Jul 10 19:11:03.188568 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.188525    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerStarted","Data":"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"}
Jul 10 19:11:03.188729 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.188590    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerStarted","Data":"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"}
Jul 10 19:11:03.188729 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.188622    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerStarted","Data":"db37d09360f14214e5dbcb0e99880b6116fbb7a9345949b83775c0f0977f3a60"}
Jul 10 19:11:03.188729 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.188656    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:03.190403 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.190384    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerStarted","Data":"edf7047cb44e881fadfda025ec71b017b7a6bb65809be435cd105c3f8b7035b6"}
Jul 10 19:11:03.190495 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.190423    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:03.190495 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.190443    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerStarted","Data":"169a1659e414395dd7c5848d8c00bdcd4805306004c8189536a8f29ffbaa754e"}
Jul 10 19:11:03.190495 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.190461    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerStarted","Data":"28ccba479235c17f548e02832e6a225093d9aff2144e6bc73e746d302f88d3d8"}
Jul 10 19:11:03.190495 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.190491    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:03.191707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.191685    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:03.208142 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.208047    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podStartSLOduration=1.208027665 podStartE2EDuration="1.208027665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:03.205396052 +0000 UTC m=+749.503747424" watchObservedRunningTime="2026-07-10 19:11:03.208027665 +0000 UTC m=+749.506379036"
Jul 10 19:11:03.223353 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:03.223309    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podStartSLOduration=1.223293215 podStartE2EDuration="1.223293215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:03.222590447 +0000 UTC m=+749.520941817" watchObservedRunningTime="2026-07-10 19:11:03.223293215 +0000 UTC m=+749.521644584"
Jul 10 19:11:04.194730 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:04.194699    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:04.194730 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:04.194704    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:04.195811 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:04.195783    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:05.061840 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:05.061798    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 10 19:11:05.199001 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:05.198960    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:05.796106 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:05.796068    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 10 19:11:08.062262 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.062203    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:11:08.119472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119438    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config\") pod \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") "
Jul 10 19:11:08.119636 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119494    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location\") pod \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") "
Jul 10 19:11:08.119636 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119535    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls\") pod \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") "
Jul 10 19:11:08.119636 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119587    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert\") pod \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") "
Jul 10 19:11:08.119636 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119614    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qk76\" (UniqueName: \"kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76\") pod \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\" (UID: \"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95\") "
Jul 10 19:11:08.119896 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119862    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" (UID: "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:11:08.119950 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119865    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" (UID: "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:08.120005 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.119984    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" (UID: "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:08.121776 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.121720    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" (UID: "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:08.121776 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.121727    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76" (OuterVolumeSpecName: "kube-api-access-4qk76") pod "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" (UID: "84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95"). InnerVolumeSpecName "kube-api-access-4qk76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:11:08.216494 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.216455    2550 generic.go:356] "Generic (PLEG): container finished" podID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerID="ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40" exitCode=0
Jul 10 19:11:08.216652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.216529    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerDied","Data":"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"}
Jul 10 19:11:08.216652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.216539    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"
Jul 10 19:11:08.216652 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.216628    2550 scope.go:117] "RemoveContainer" containerID="65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"
Jul 10 19:11:08.216778 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.216617    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z" event={"ID":"84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95","Type":"ContainerDied","Data":"3f9fdf58e6781e7a8a159af66158843430491e790295c1d2f53a10e9ac3222ca"}
Jul 10 19:11:08.220239 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.220199    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-cabundle-cert\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.220239 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.220228    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qk76\" (UniqueName: \"kubernetes.io/projected/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-api-access-4qk76\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.220424 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.220245    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.220424 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.220259    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-kserve-provision-location\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.220424 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.220273    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:11:08.227102 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.227084    2550 scope.go:117] "RemoveContainer" containerID="ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"
Jul 10 19:11:08.234622 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.234603    2550 scope.go:117] "RemoveContainer" containerID="1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"
Jul 10 19:11:08.241610 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.241583    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:11:08.242236 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.242170    2550 scope.go:117] "RemoveContainer" containerID="65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"
Jul 10 19:11:08.242494 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:11:08.242471    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb\": container with ID starting with 65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb not found: ID does not exist" containerID="65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"
Jul 10 19:11:08.242581 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.242503    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb"} err="failed to get container status \"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb\": rpc error: code = NotFound desc = could not find container \"65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb\": container with ID starting with 65d3aa29c2e77be2863247cd11772eca59decd1cd85553f43e627f63b04754cb not found: ID does not exist"
Jul 10 19:11:08.242581 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.242535    2550 scope.go:117] "RemoveContainer" containerID="ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"
Jul 10 19:11:08.243010 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:11:08.242990    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40\": container with ID starting with ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40 not found: ID does not exist" containerID="ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"
Jul 10 19:11:08.243085 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.243014    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40"} err="failed to get container status \"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40\": rpc error: code = NotFound desc = could not find container \"ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40\": container with ID starting with ecb66c60099b1969ea0e4b4357ea30477feaf219674e17a851e005dbe27bfb40 not found: ID does not exist"
Jul 10 19:11:08.243085 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.243029    2550 scope.go:117] "RemoveContainer" containerID="1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"
Jul 10 19:11:08.243284 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:11:08.243262    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156\": container with ID starting with 1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156 not found: ID does not exist" containerID="1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"
Jul 10 19:11:08.243341 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.243290    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156"} err="failed to get container status \"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156\": rpc error: code = NotFound desc = could not find container \"1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156\": container with ID starting with 1a243b6c77f5f8fc40e2059b7aaba4c6e7fda57acb1ae6d19aa263ca8666f156 not found: ID does not exist"
Jul 10 19:11:08.243790 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.243772    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-cb544c5dc-5l29z"]
Jul 10 19:11:08.274610 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:08.274584    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" path="/var/lib/kubelet/pods/84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95/volumes"
Jul 10 19:11:09.198606 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:09.198576    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:09.199068 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:09.199006    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:10.203983 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:10.203954    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:11:10.204598 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:10.204569    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:13.049477 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:13.049452    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:11:15.061940 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:15.061906    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:11:19.200005 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:19.199969    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:20.204579 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:20.204534    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:29.199811 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:29.199716    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:30.204413 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:30.204379    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:39.199134 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:39.199091    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:11:40.204846 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:40.204790    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:11:49.200451 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:49.200422    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:11:50.205090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:11:50.205061    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:19:41.153181 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.153144    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:19:41.155450 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.153358    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" containerID="cri-o://174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677" gracePeriod=30
Jul 10 19:19:41.155450 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.153414    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kube-rbac-proxy" containerID="cri-o://ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8" gracePeriod=30
Jul 10 19:19:41.200106 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.200074    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:19:41.200310 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.200289    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container" containerID="cri-o://2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e" gracePeriod=30
Jul 10 19:19:41.200415 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.200369    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kube-rbac-proxy" containerID="cri-o://49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d" gracePeriod=30
Jul 10 19:19:41.275449 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.275416    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:19:41.275940 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.275924    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="storage-initializer"
Jul 10 19:19:41.276016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.275971    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="storage-initializer"
Jul 10 19:19:41.276016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.275996    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container"
Jul 10 19:19:41.276016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.276004    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container"
Jul 10 19:19:41.276016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.276015    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kube-rbac-proxy"
Jul 10 19:19:41.276214 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.276024    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kube-rbac-proxy"
Jul 10 19:19:41.276214 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.276097    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kserve-container"
Jul 10 19:19:41.276214 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.276110    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="84fdc5e7-4a67-4ee2-8b3f-3fa819d62c95" containerName="kube-rbac-proxy"
Jul 10 19:19:41.279723 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.279698    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.285469 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.285443    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3dba1-predictor-serving-cert\""
Jul 10 19:19:41.285804 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.285787    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3dba1-kube-rbac-proxy-sar-config\""
Jul 10 19:19:41.291326 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.291303    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:19:41.332648 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.332613    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:19:41.336419 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.336389    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.338964 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.338936    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3dba1-predictor-serving-cert\""
Jul 10 19:19:41.339074 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.338940    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3dba1-kube-rbac-proxy-sar-config\""
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343577    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343620    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343657    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343697    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343776    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbst\" (UniqueName: \"kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.344132 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.343813    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf2r\" (UniqueName: \"kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.345411 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.345386    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:19:41.445089 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.444995    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.445089 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445048    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.445089 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445084    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:41.445167    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-serving-cert: secret "error-404-isvc-3dba1-predictor-serving-cert" not found
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:41.445225    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-serving-cert: secret "success-200-isvc-3dba1-predictor-serving-cert" not found
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:41.445258    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls podName:4973cc2d-3c76-450c-887f-dc31c7e95859 nodeName:}" failed. No retries permitted until 2026-07-10 19:19:41.945238226 +0000 UTC m=+1268.243589599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls") pod "error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" (UID: "4973cc2d-3c76-450c-887f-dc31c7e95859") : secret "error-404-isvc-3dba1-predictor-serving-cert" not found
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:41.445302    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls podName:0a8e24b7-7139-43b6-90b9-20860369b7bf nodeName:}" failed. No retries permitted until 2026-07-10 19:19:41.94528384 +0000 UTC m=+1268.243635190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls") pod "success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" (UID: "0a8e24b7-7139-43b6-90b9-20860369b7bf") : secret "success-200-isvc-3dba1-predictor-serving-cert" not found
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445230    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.445365 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445355    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbst\" (UniqueName: \"kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.445668 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445392    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf2r\" (UniqueName: \"kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.445860 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.445835    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.446030 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.446009    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.453798 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.453776    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbst\" (UniqueName: \"kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.454045 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.454026    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf2r\" (UniqueName: \"kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.948806 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.948774    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:41.948975 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.948816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.951389 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.951356    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") pod \"success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") " pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:41.951496 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:41.951396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") pod \"error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") " pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:42.142764 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.142713    2550 generic.go:356] "Generic (PLEG): container finished" podID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerID="ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8" exitCode=2
Jul 10 19:19:42.142936 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.142788    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerDied","Data":"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"}
Jul 10 19:19:42.144627 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.144599    2550 generic.go:356] "Generic (PLEG): container finished" podID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerID="49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d" exitCode=2
Jul 10 19:19:42.144719 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.144663    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerDied","Data":"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"}
Jul 10 19:19:42.192616 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.192584    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:42.248909 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.248839    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:42.342022 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.341995    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:19:42.345708 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:19:42.345679    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8e24b7_7139_43b6_90b9_20860369b7bf.slice/crio-d4d3131a9a0a71f7dfa6463057aa0a93cd7bd673e6cc0631e0ca75a402dba2ed WatchSource:0}: Error finding container d4d3131a9a0a71f7dfa6463057aa0a93cd7bd673e6cc0631e0ca75a402dba2ed: Status 404 returned error can't find the container with id d4d3131a9a0a71f7dfa6463057aa0a93cd7bd673e6cc0631e0ca75a402dba2ed
Jul 10 19:19:42.393756 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:42.393689    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:19:42.395808 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:19:42.395789    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4973cc2d_3c76_450c_887f_dc31c7e95859.slice/crio-72cab0bdfd727c969bf15ce3b35ae6810928a6af4c4e52cb6a07fa4b76025e1f WatchSource:0}: Error finding container 72cab0bdfd727c969bf15ce3b35ae6810928a6af4c4e52cb6a07fa4b76025e1f: Status 404 returned error can't find the container with id 72cab0bdfd727c969bf15ce3b35ae6810928a6af4c4e52cb6a07fa4b76025e1f
Jul 10 19:19:43.044728 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.044684    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 10 19:19:43.048970 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.048948    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 10 19:19:43.150936 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.150901    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerStarted","Data":"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"}
Jul 10 19:19:43.151108 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.150976    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerStarted","Data":"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"}
Jul 10 19:19:43.151108 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.151010    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerStarted","Data":"72cab0bdfd727c969bf15ce3b35ae6810928a6af4c4e52cb6a07fa4b76025e1f"}
Jul 10 19:19:43.151108 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.151045    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:43.152712 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.152689    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerStarted","Data":"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"}
Jul 10 19:19:43.152853 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.152754    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerStarted","Data":"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"}
Jul 10 19:19:43.152853 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.152774    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerStarted","Data":"d4d3131a9a0a71f7dfa6463057aa0a93cd7bd673e6cc0631e0ca75a402dba2ed"}
Jul 10 19:19:43.152967 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.152869    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:43.152967 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.152908    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:43.154168 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.154143    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:19:43.169400 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.169345    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podStartSLOduration=2.16933225 podStartE2EDuration="2.16933225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:43.168075264 +0000 UTC m=+1269.466426634" watchObservedRunningTime="2026-07-10 19:19:43.16933225 +0000 UTC m=+1269.467683670"
Jul 10 19:19:43.185752 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:43.185690    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podStartSLOduration=2.18567335 podStartE2EDuration="2.18567335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:43.184009615 +0000 UTC m=+1269.482360985" watchObservedRunningTime="2026-07-10 19:19:43.18567335 +0000 UTC m=+1269.484024721"
Jul 10 19:19:44.156694 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.156655    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:19:44.157095 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.156755    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:44.157891 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.157865    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:19:44.499488 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.499464    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:19:44.570321 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.570294    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config\") pod \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") "
Jul 10 19:19:44.570453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.570362    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-644zp\" (UniqueName: \"kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp\") pod \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") "
Jul 10 19:19:44.570453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.570440    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls\") pod \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\" (UID: \"68099bf1-5bb6-470a-b8f1-5e3a611f0679\") "
Jul 10 19:19:44.570614 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.570590    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68099bf1-5bb6-470a-b8f1-5e3a611f0679" (UID: "68099bf1-5bb6-470a-b8f1-5e3a611f0679"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:19:44.572384 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.572353    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp" (OuterVolumeSpecName: "kube-api-access-644zp") pod "68099bf1-5bb6-470a-b8f1-5e3a611f0679" (UID: "68099bf1-5bb6-470a-b8f1-5e3a611f0679"). InnerVolumeSpecName "kube-api-access-644zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:19:44.572514 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.572494    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68099bf1-5bb6-470a-b8f1-5e3a611f0679" (UID: "68099bf1-5bb6-470a-b8f1-5e3a611f0679"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:19:44.671571 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.671515    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644zp\" (UniqueName: \"kubernetes.io/projected/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-api-access-644zp\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:44.671571 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.671538    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68099bf1-5bb6-470a-b8f1-5e3a611f0679-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:44.671571 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:44.671568    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68099bf1-5bb6-470a-b8f1-5e3a611f0679-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.037040 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.037016    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:19:45.074486 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.074457    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config\") pod \"69e78f05-d420-4352-bb49-9336e87fa5e8\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") "
Jul 10 19:19:45.074637 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.074543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") pod \"69e78f05-d420-4352-bb49-9336e87fa5e8\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") "
Jul 10 19:19:45.074637 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.074568    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvqx9\" (UniqueName: \"kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9\") pod \"69e78f05-d420-4352-bb49-9336e87fa5e8\" (UID: \"69e78f05-d420-4352-bb49-9336e87fa5e8\") "
Jul 10 19:19:45.074872 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.074849    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69e78f05-d420-4352-bb49-9336e87fa5e8" (UID: "69e78f05-d420-4352-bb49-9336e87fa5e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:19:45.076623 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.076598    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69e78f05-d420-4352-bb49-9336e87fa5e8" (UID: "69e78f05-d420-4352-bb49-9336e87fa5e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:19:45.076712 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.076657    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9" (OuterVolumeSpecName: "kube-api-access-zvqx9") pod "69e78f05-d420-4352-bb49-9336e87fa5e8" (UID: "69e78f05-d420-4352-bb49-9336e87fa5e8"). InnerVolumeSpecName "kube-api-access-zvqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:19:45.162645 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.162615    2550 generic.go:356] "Generic (PLEG): container finished" podID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerID="174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677" exitCode=0
Jul 10 19:19:45.163073 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.162664    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"
Jul 10 19:19:45.163073 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.162704    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerDied","Data":"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"}
Jul 10 19:19:45.163073 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.162828    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm" event={"ID":"68099bf1-5bb6-470a-b8f1-5e3a611f0679","Type":"ContainerDied","Data":"bcc380f7b9a4189e7299a3375a63319834560da69a19951202a10bedb47dd14a"}
Jul 10 19:19:45.163073 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.162862    2550 scope.go:117] "RemoveContainer" containerID="ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"
Jul 10 19:19:45.164575 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.164550    2550 generic.go:356] "Generic (PLEG): container finished" podID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerID="2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e" exitCode=0
Jul 10 19:19:45.164675 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.164615    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"
Jul 10 19:19:45.164675 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.164641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerDied","Data":"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"}
Jul 10 19:19:45.164779 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.164703    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr" event={"ID":"69e78f05-d420-4352-bb49-9336e87fa5e8","Type":"ContainerDied","Data":"5b10a3084ee5403f8be28885198b3917647015fc1c3ce2ef737e7826e716c520"}
Jul 10 19:19:45.165243 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.165209    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:19:45.171697 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.171676    2550 scope.go:117] "RemoveContainer" containerID="174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"
Jul 10 19:19:45.175849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.175246    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69e78f05-d420-4352-bb49-9336e87fa5e8-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.175849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.175266    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqx9\" (UniqueName: \"kubernetes.io/projected/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-api-access-zvqx9\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.175849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.175280    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69e78f05-d420-4352-bb49-9336e87fa5e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.181207 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.181187    2550 scope.go:117] "RemoveContainer" containerID="ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"
Jul 10 19:19:45.181451 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:45.181434    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8\": container with ID starting with ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8 not found: ID does not exist" containerID="ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"
Jul 10 19:19:45.181505 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.181458    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8"} err="failed to get container status \"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8\": rpc error: code = NotFound desc = could not find container \"ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8\": container with ID starting with ac2a8127f2b9c621cc05fb45188e56de14a85b7ad84cfea5f5dba5fb2187c4c8 not found: ID does not exist"
Jul 10 19:19:45.181505 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.181471    2550 scope.go:117] "RemoveContainer" containerID="174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"
Jul 10 19:19:45.181708 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:45.181690    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677\": container with ID starting with 174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677 not found: ID does not exist" containerID="174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"
Jul 10 19:19:45.181845 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.181722    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677"} err="failed to get container status \"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677\": rpc error: code = NotFound desc = could not find container \"174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677\": container with ID starting with 174e664d3e2326a3c89100432de0939783a5c466d68c2e900680bccb3e034677 not found: ID does not exist"
Jul 10 19:19:45.181845 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.181775    2550 scope.go:117] "RemoveContainer" containerID="49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"
Jul 10 19:19:45.188991 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.188973    2550 scope.go:117] "RemoveContainer" containerID="2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"
Jul 10 19:19:45.189074 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.188993    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:19:45.194806 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.194777    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4aae5-predictor-f8cd6867f-plzsm"]
Jul 10 19:19:45.196554 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.196539    2550 scope.go:117] "RemoveContainer" containerID="49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"
Jul 10 19:19:45.196831 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:45.196814    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d\": container with ID starting with 49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d not found: ID does not exist" containerID="49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"
Jul 10 19:19:45.196887 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.196836    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d"} err="failed to get container status \"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d\": rpc error: code = NotFound desc = could not find container \"49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d\": container with ID starting with 49657ce964d7d4964beca9f1ceb79cd2cb47bc431be337c896f839aa9950bc9d not found: ID does not exist"
Jul 10 19:19:45.196887 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.196850    2550 scope.go:117] "RemoveContainer" containerID="2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"
Jul 10 19:19:45.197102 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:19:45.197085    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e\": container with ID starting with 2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e not found: ID does not exist" containerID="2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"
Jul 10 19:19:45.197154 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.197110    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e"} err="failed to get container status \"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e\": rpc error: code = NotFound desc = could not find container \"2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e\": container with ID starting with 2552bad1ad1258692a7454bb08d2f521d57426133c981305d8e2491ffde9243e not found: ID does not exist"
Jul 10 19:19:45.205531 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.205510    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:19:45.206877 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:45.206858    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4aae5-predictor-788b8dfbc4-7ffrr"]
Jul 10 19:19:46.275790 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:46.275761    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" path="/var/lib/kubelet/pods/68099bf1-5bb6-470a-b8f1-5e3a611f0679/volumes"
Jul 10 19:19:46.276170 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:46.276157    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" path="/var/lib/kubelet/pods/69e78f05-d420-4352-bb49-9336e87fa5e8/volumes"
Jul 10 19:19:49.160919 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:49.160893    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:19:49.161462 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:49.161438    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:19:50.169937 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:50.169902    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:19:50.170340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:50.170316    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:19:59.161672 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:19:59.161635    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:20:00.171057 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:00.171018    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:20:09.162165 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:09.162127    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:20:10.171058 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:10.171017    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:20:17.263281 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.263243    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:20:17.263763 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.263510    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" containerID="cri-o://e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44" gracePeriod=30
Jul 10 19:20:17.263763 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.263550    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kube-rbac-proxy" containerID="cri-o://fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98" gracePeriod=30
Jul 10 19:20:17.334482 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.334450    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:20:17.334700 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.334667    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" containerID="cri-o://169a1659e414395dd7c5848d8c00bdcd4805306004c8189536a8f29ffbaa754e" gracePeriod=30
Jul 10 19:20:17.334868 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.334751    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kube-rbac-proxy" containerID="cri-o://edf7047cb44e881fadfda025ec71b017b7a6bb65809be435cd105c3f8b7035b6" gracePeriod=30
Jul 10 19:20:17.353598 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353565    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:20:17.353943 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353929    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kube-rbac-proxy"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353945    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kube-rbac-proxy"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353961    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353966    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353972    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kube-rbac-proxy"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353977    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kube-rbac-proxy"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353986    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container"
Jul 10 19:20:17.354006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.353991    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container"
Jul 10 19:20:17.354353 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.354040    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kserve-container"
Jul 10 19:20:17.354353 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.354050    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kube-rbac-proxy"
Jul 10 19:20:17.354353 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.354056    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e78f05-d420-4352-bb49-9336e87fa5e8" containerName="kserve-container"
Jul 10 19:20:17.354353 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.354063    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="68099bf1-5bb6-470a-b8f1-5e3a611f0679" containerName="kube-rbac-proxy"
Jul 10 19:20:17.357065 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.357047    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.359410 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.359391    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d30ee-predictor-serving-cert\""
Jul 10 19:20:17.359664 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.359646    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d30ee-kube-rbac-proxy-sar-config\""
Jul 10 19:20:17.367302 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.367278    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:20:17.442578 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.442553    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.442695 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.442600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfgz\" (UniqueName: \"kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.442695 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.442621    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.451114 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.451088    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:20:17.454387 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.454369    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.456655 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.456631    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d30ee-predictor-serving-cert\""
Jul 10 19:20:17.456822 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.456660    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d30ee-kube-rbac-proxy-sar-config\""
Jul 10 19:20:17.464339 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.464316    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:20:17.543666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543587    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.543666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543667    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcc9\" (UniqueName: \"kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543708    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfgz\" (UniqueName: \"kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543749    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:17.543769    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-serving-cert: secret "success-200-isvc-d30ee-predictor-serving-cert" not found
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.543842    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.543902 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:17.543856    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls podName:17b39e35-525f-4039-88b4-50169241e202 nodeName:}" failed. No retries permitted until 2026-07-10 19:20:18.04383598 +0000 UTC m=+1304.342187348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls") pod "success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" (UID: "17b39e35-525f-4039-88b4-50169241e202") : secret "success-200-isvc-d30ee-predictor-serving-cert" not found
Jul 10 19:20:17.544383 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.544363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.554362 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.554330    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfgz\" (UniqueName: \"kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:17.644472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.644440    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.644667 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.644504    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.644667 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.644524    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcc9\" (UniqueName: \"kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.645135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.645113    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.647037 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.647018    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.652789 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.652766    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcc9\" (UniqueName: \"kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9\") pod \"error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") " pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.765775 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.765719    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:17.898368 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:17.898346    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:20:17.900375 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:20:17.900342    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4d78ba_478a_41a2_bb7c_77744b61bb2f.slice/crio-956cee25a4b36921300bcdf9d675afd2250094ee2050bac5ebf866286a025709 WatchSource:0}: Error finding container 956cee25a4b36921300bcdf9d675afd2250094ee2050bac5ebf866286a025709: Status 404 returned error can't find the container with id 956cee25a4b36921300bcdf9d675afd2250094ee2050bac5ebf866286a025709
Jul 10 19:20:18.048179 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.048147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:18.050437 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.050413    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") pod \"success-200-isvc-d30ee-predictor-585b9dc857-dqrvt\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") " pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:18.269855 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.269810    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:18.310392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.308453    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerStarted","Data":"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"}
Jul 10 19:20:18.310392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.308530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerStarted","Data":"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"}
Jul 10 19:20:18.310392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.308556    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerStarted","Data":"956cee25a4b36921300bcdf9d675afd2250094ee2050bac5ebf866286a025709"}
Jul 10 19:20:18.310392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.309856    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:18.310392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.309912    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:18.311015 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.310972    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:18.312397 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.312372    2550 generic.go:356] "Generic (PLEG): container finished" podID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerID="fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98" exitCode=2
Jul 10 19:20:18.312512 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.312445    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerDied","Data":"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"}
Jul 10 19:20:18.314673 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.314649    2550 generic.go:356] "Generic (PLEG): container finished" podID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerID="edf7047cb44e881fadfda025ec71b017b7a6bb65809be435cd105c3f8b7035b6" exitCode=2
Jul 10 19:20:18.314781 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.314686    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerDied","Data":"edf7047cb44e881fadfda025ec71b017b7a6bb65809be435cd105c3f8b7035b6"}
Jul 10 19:20:18.328827 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.328766    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podStartSLOduration=1.328725568 podStartE2EDuration="1.328725568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:18.32639525 +0000 UTC m=+1304.624746619" watchObservedRunningTime="2026-07-10 19:20:18.328725568 +0000 UTC m=+1304.627076938"
Jul 10 19:20:18.405625 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:18.405592    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:20:18.407928 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:20:18.407892    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b39e35_525f_4039_88b4_50169241e202.slice/crio-7191b17bf466665eb3bce77d8dfbc15a3a897bd1e0dbeae26741f31b26d22bee WatchSource:0}: Error finding container 7191b17bf466665eb3bce77d8dfbc15a3a897bd1e0dbeae26741f31b26d22bee: Status 404 returned error can't find the container with id 7191b17bf466665eb3bce77d8dfbc15a3a897bd1e0dbeae26741f31b26d22bee
Jul 10 19:20:19.162265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.162229    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 10 19:20:19.194929 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.194893    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 10 19:20:19.199158 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.199133    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 10 19:20:19.321153 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.321109    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerStarted","Data":"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"}
Jul 10 19:20:19.321604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.321179    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerStarted","Data":"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"}
Jul 10 19:20:19.321604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.321202    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerStarted","Data":"7191b17bf466665eb3bce77d8dfbc15a3a897bd1e0dbeae26741f31b26d22bee"}
Jul 10 19:20:19.321604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.321440    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:19.321604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.321496    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:19.340234 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:19.340195    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podStartSLOduration=2.340183963 podStartE2EDuration="2.340183963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:19.339089985 +0000 UTC m=+1305.637441365" watchObservedRunningTime="2026-07-10 19:20:19.340183963 +0000 UTC m=+1305.638535333"
Jul 10 19:20:20.171247 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.171208    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 10 19:20:20.199830 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.199792    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 10 19:20:20.205271 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.205235    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 10 19:20:20.326450 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.326402    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:20.326945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.326513    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:20.327790 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:20.327768    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:21.225072 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.225049    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:20:21.331833 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.331794    2550 generic.go:356] "Generic (PLEG): container finished" podID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerID="e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44" exitCode=0
Jul 10 19:20:21.332280 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.331869    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"
Jul 10 19:20:21.332280 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.331880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerDied","Data":"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"}
Jul 10 19:20:21.332280 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.331990    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52" event={"ID":"0d6e9d06-c511-4bfe-9373-701578db33e5","Type":"ContainerDied","Data":"db37d09360f14214e5dbcb0e99880b6116fbb7a9345949b83775c0f0977f3a60"}
Jul 10 19:20:21.332280 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.332033    2550 scope.go:117] "RemoveContainer" containerID="fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"
Jul 10 19:20:21.333705 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.333685    2550 generic.go:356] "Generic (PLEG): container finished" podID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerID="169a1659e414395dd7c5848d8c00bdcd4805306004c8189536a8f29ffbaa754e" exitCode=0
Jul 10 19:20:21.333825 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.333764    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerDied","Data":"169a1659e414395dd7c5848d8c00bdcd4805306004c8189536a8f29ffbaa754e"}
Jul 10 19:20:21.334252 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.334230    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:21.340211 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.340202    2550 scope.go:117] "RemoveContainer" containerID="e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"
Jul 10 19:20:21.347386 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.347370    2550 scope.go:117] "RemoveContainer" containerID="fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"
Jul 10 19:20:21.347663 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:21.347644    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98\": container with ID starting with fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98 not found: ID does not exist" containerID="fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"
Jul 10 19:20:21.347708 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.347672    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98"} err="failed to get container status \"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98\": rpc error: code = NotFound desc = could not find container \"fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98\": container with ID starting with fa323ae41de675b08e6f8ec87f2cce6746db6b2938e3bf0c1a6ba817ba451b98 not found: ID does not exist"
Jul 10 19:20:21.347708 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.347689    2550 scope.go:117] "RemoveContainer" containerID="e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"
Jul 10 19:20:21.347929 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:21.347911    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44\": container with ID starting with e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44 not found: ID does not exist" containerID="e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"
Jul 10 19:20:21.347969 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.347935    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44"} err="failed to get container status \"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44\": rpc error: code = NotFound desc = could not find container \"e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44\": container with ID starting with e6858a6a6e395f2c2e5afa941ba0569954b67f8422540f182fdd6f9d2ca42f44 not found: ID does not exist"
Jul 10 19:20:21.376190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.376168    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls\") pod \"0d6e9d06-c511-4bfe-9373-701578db33e5\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") "
Jul 10 19:20:21.376261 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.376206    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqf58\" (UniqueName: \"kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58\") pod \"0d6e9d06-c511-4bfe-9373-701578db33e5\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") "
Jul 10 19:20:21.376261 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.376237    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config\") pod \"0d6e9d06-c511-4bfe-9373-701578db33e5\" (UID: \"0d6e9d06-c511-4bfe-9373-701578db33e5\") "
Jul 10 19:20:21.376583 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.376560    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d6e9d06-c511-4bfe-9373-701578db33e5" (UID: "0d6e9d06-c511-4bfe-9373-701578db33e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:21.378345 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.378322    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d6e9d06-c511-4bfe-9373-701578db33e5" (UID: "0d6e9d06-c511-4bfe-9373-701578db33e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:21.378426 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.378324    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58" (OuterVolumeSpecName: "kube-api-access-zqf58") pod "0d6e9d06-c511-4bfe-9373-701578db33e5" (UID: "0d6e9d06-c511-4bfe-9373-701578db33e5"). InnerVolumeSpecName "kube-api-access-zqf58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:21.477681 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.477585    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d6e9d06-c511-4bfe-9373-701578db33e5-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:21.478393 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.478366    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf58\" (UniqueName: \"kubernetes.io/projected/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-api-access-zqf58\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:21.478393 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.478394    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d6e9d06-c511-4bfe-9373-701578db33e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:21.578655 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.578631    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:20:21.661356 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.661327    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:20:21.667058 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.667036    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31f26-predictor-7599c98b7f-ctz52"]
Jul 10 19:20:21.679077 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.679057    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config\") pod \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") "
Jul 10 19:20:21.679160 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.679132    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9d4r\" (UniqueName: \"kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r\") pod \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") "
Jul 10 19:20:21.679223 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.679158    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls\") pod \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\" (UID: \"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23\") "
Jul 10 19:20:21.679447 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.679424    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" (UID: "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:21.681272 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.681241    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r" (OuterVolumeSpecName: "kube-api-access-r9d4r") pod "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" (UID: "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23"). InnerVolumeSpecName "kube-api-access-r9d4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:21.681362 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.681272    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" (UID: "fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:21.780547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.780467    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:21.780547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.780493    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d4r\" (UniqueName: \"kubernetes.io/projected/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-kube-api-access-r9d4r\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:21.780547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:21.780503    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:22.274900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.274865    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" path="/var/lib/kubelet/pods/0d6e9d06-c511-4bfe-9373-701578db33e5/volumes"
Jul 10 19:20:22.341226 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.341192    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r" event={"ID":"fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23","Type":"ContainerDied","Data":"28ccba479235c17f548e02832e6a225093d9aff2144e6bc73e746d302f88d3d8"}
Jul 10 19:20:22.341607 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.341231    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"
Jul 10 19:20:22.341607 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.341261    2550 scope.go:117] "RemoveContainer" containerID="edf7047cb44e881fadfda025ec71b017b7a6bb65809be435cd105c3f8b7035b6"
Jul 10 19:20:22.349493 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.349474    2550 scope.go:117] "RemoveContainer" containerID="169a1659e414395dd7c5848d8c00bdcd4805306004c8189536a8f29ffbaa754e"
Jul 10 19:20:22.359519 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.359497    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:20:22.362929 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:22.362910    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31f26-predictor-c7f49fbf8-gxg8r"]
Jul 10 19:20:24.274236 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:24.274197    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" path="/var/lib/kubelet/pods/fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23/volumes"
Jul 10 19:20:25.330658 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:25.330626    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:20:25.331198 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:25.331173    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:26.338642 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:26.338617    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:20:26.339178 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:26.339150    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:29.162497 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:29.162460    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:20:30.170911 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:30.170877    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:20:35.332038 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:35.331997    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:36.339705 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:36.339668    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:45.331612 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:45.331570    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:46.339931 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:46.339890    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:51.461978 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.461947    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:20:51.462430 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.462194    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container" containerID="cri-o://a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960" gracePeriod=30
Jul 10 19:20:51.462430 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.462234    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kube-rbac-proxy" containerID="cri-o://649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f" gracePeriod=30
Jul 10 19:20:51.512837 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.512807    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:20:51.513089 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.513059    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kube-rbac-proxy" containerID="cri-o://6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa" gracePeriod=30
Jul 10 19:20:51.513229 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.513053    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container" containerID="cri-o://6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c" gracePeriod=30
Jul 10 19:20:51.589290 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589257    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:20:51.589801 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589785    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container"
Jul 10 19:20:51.589848 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589805    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container"
Jul 10 19:20:51.589848 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589818    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kube-rbac-proxy"
Jul 10 19:20:51.589848 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589827    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kube-rbac-proxy"
Jul 10 19:20:51.589848 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589844    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container"
Jul 10 19:20:51.589968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589852    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container"
Jul 10 19:20:51.589968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589871    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kube-rbac-proxy"
Jul 10 19:20:51.589968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589879    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kube-rbac-proxy"
Jul 10 19:20:51.589968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589957    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kserve-container"
Jul 10 19:20:51.590081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589970    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kube-rbac-proxy"
Jul 10 19:20:51.590081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589979    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9cfa8c-7f6a-4a51-80a8-5fe6fee09f23" containerName="kserve-container"
Jul 10 19:20:51.590081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.589990    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d6e9d06-c511-4bfe-9373-701578db33e5" containerName="kube-rbac-proxy"
Jul 10 19:20:51.593184 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.593157    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.595504 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.595485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d232d-predictor-serving-cert\""
Jul 10 19:20:51.595781 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.595764    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d232d-kube-rbac-proxy-sar-config\""
Jul 10 19:20:51.602075 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.601880    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqz4h\" (UniqueName: \"kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.602075 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.601925    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.602075 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.601967    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.603984 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.603916    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:20:51.673004 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.672966    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:20:51.676364 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.676346    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.678651 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.678633    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d232d-predictor-serving-cert\""
Jul 10 19:20:51.678780 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.678672    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d232d-kube-rbac-proxy-sar-config\""
Jul 10 19:20:51.686430 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.686407    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:20:51.702798 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.702774    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqz4h\" (UniqueName: \"kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.702917 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.702824    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.702917 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.702862    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.703033 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.702920    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.703033 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.702964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lgr\" (UniqueName: \"kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.703033 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.703014    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.703845 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.703825    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.707224 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.706558    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.715503 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.714540    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqz4h\" (UniqueName: \"kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h\") pod \"success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") " pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:51.803496 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.803445    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.803689 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.803529    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lgr\" (UniqueName: \"kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.803689 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.803578    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.803837 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:51.803720    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d232d-predictor-serving-cert: secret "error-404-isvc-d232d-predictor-serving-cert" not found
Jul 10 19:20:51.803837 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:51.803812    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls podName:1d8b4dd5-a9be-4ded-8f51-c3f498f541b0 nodeName:}" failed. No retries permitted until 2026-07-10 19:20:52.303789424 +0000 UTC m=+1338.602140772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls") pod "error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" (UID: "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0") : secret "error-404-isvc-d232d-predictor-serving-cert" not found
Jul 10 19:20:51.804812 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.804765    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.813023 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.812989    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lgr\" (UniqueName: \"kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:51.904850 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:51.904801    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:52.042577 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.042544    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:20:52.043966 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:20:52.043939    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e9d3ba_f634_4ed4_b81e_d82a66f50c3c.slice/crio-68925ac1326cd914b0c1f669cec0c9257f8e7efe5e75dd9e27c9f8562158a4d3 WatchSource:0}: Error finding container 68925ac1326cd914b0c1f669cec0c9257f8e7efe5e75dd9e27c9f8562158a4d3: Status 404 returned error can't find the container with id 68925ac1326cd914b0c1f669cec0c9257f8e7efe5e75dd9e27c9f8562158a4d3
Jul 10 19:20:52.312602 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.312496    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:52.315182 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.315161    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") pod \"error-404-isvc-d232d-predictor-6d6f664f96-bzwzh\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") " pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:52.471763 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.471713    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerStarted","Data":"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"}
Jul 10 19:20:52.472193 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.471798    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:52.472193 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.471825    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerStarted","Data":"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"}
Jul 10 19:20:52.472193 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.471856    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerStarted","Data":"68925ac1326cd914b0c1f669cec0c9257f8e7efe5e75dd9e27c9f8562158a4d3"}
Jul 10 19:20:52.472193 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.471956    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:52.473304 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.473276    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:20:52.473803 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.473780    2550 generic.go:356] "Generic (PLEG): container finished" podID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerID="649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f" exitCode=2
Jul 10 19:20:52.473917 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.473845    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerDied","Data":"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"}
Jul 10 19:20:52.475674 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.475654    2550 generic.go:356] "Generic (PLEG): container finished" podID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerID="6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa" exitCode=2
Jul 10 19:20:52.475777 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.475685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerDied","Data":"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"}
Jul 10 19:20:52.587704 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.587628    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:52.736914 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.736864    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podStartSLOduration=1.736846431 podStartE2EDuration="1.736846431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:52.522665499 +0000 UTC m=+1338.821016880" watchObservedRunningTime="2026-07-10 19:20:52.736846431 +0000 UTC m=+1339.035197803"
Jul 10 19:20:52.738112 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:52.738091    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:20:52.741358 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:20:52.741333    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8b4dd5_a9be_4ded_8f51_c3f498f541b0.slice/crio-d264eb207ac47fa31e4ad43de850df615814019606a337de982730e26edd6f72 WatchSource:0}: Error finding container d264eb207ac47fa31e4ad43de850df615814019606a337de982730e26edd6f72: Status 404 returned error can't find the container with id d264eb207ac47fa31e4ad43de850df615814019606a337de982730e26edd6f72
Jul 10 19:20:53.481662 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:53.481619    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerStarted","Data":"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"}
Jul 10 19:20:53.482022 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:53.481688    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerStarted","Data":"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"}
Jul 10 19:20:53.482022 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:53.481717    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerStarted","Data":"d264eb207ac47fa31e4ad43de850df615814019606a337de982730e26edd6f72"}
Jul 10 19:20:53.482022 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:53.481982    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:20:53.501661 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:53.501593    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podStartSLOduration=2.501571812 podStartE2EDuration="2.501571812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:20:53.499966136 +0000 UTC m=+1339.798317507" watchObservedRunningTime="2026-07-10 19:20:53.501571812 +0000 UTC m=+1339.799923181"
Jul 10 19:20:54.157638 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:54.157600    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 10 19:20:54.486227 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:54.486133    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:54.486227 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:54.486199    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:20:54.487422 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:54.487393    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:20:55.116937 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.116913    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:20:55.139217 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.139195    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") pod \"4973cc2d-3c76-450c-887f-dc31c7e95859\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") "
Jul 10 19:20:55.139366 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.139226    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxf2r\" (UniqueName: \"kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r\") pod \"4973cc2d-3c76-450c-887f-dc31c7e95859\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") "
Jul 10 19:20:55.139366 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.139259    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config\") pod \"4973cc2d-3c76-450c-887f-dc31c7e95859\" (UID: \"4973cc2d-3c76-450c-887f-dc31c7e95859\") "
Jul 10 19:20:55.139701 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.139669    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4973cc2d-3c76-450c-887f-dc31c7e95859" (UID: "4973cc2d-3c76-450c-887f-dc31c7e95859"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:55.141438 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.141406    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4973cc2d-3c76-450c-887f-dc31c7e95859" (UID: "4973cc2d-3c76-450c-887f-dc31c7e95859"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:55.141527 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.141494    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r" (OuterVolumeSpecName: "kube-api-access-lxf2r") pod "4973cc2d-3c76-450c-887f-dc31c7e95859" (UID: "4973cc2d-3c76-450c-887f-dc31c7e95859"). InnerVolumeSpecName "kube-api-access-lxf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:55.240726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.240698    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4973cc2d-3c76-450c-887f-dc31c7e95859-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:55.240726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.240723    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxf2r\" (UniqueName: \"kubernetes.io/projected/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-api-access-lxf2r\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:55.240726 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.240743    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4973cc2d-3c76-450c-887f-dc31c7e95859-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:55.331107 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.331072    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 10 19:20:55.351633 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.351610    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:20:55.441694 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.441664    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnbst\" (UniqueName: \"kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst\") pod \"0a8e24b7-7139-43b6-90b9-20860369b7bf\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") "
Jul 10 19:20:55.441849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.441715    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") pod \"0a8e24b7-7139-43b6-90b9-20860369b7bf\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") "
Jul 10 19:20:55.441849 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.441788    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config\") pod \"0a8e24b7-7139-43b6-90b9-20860369b7bf\" (UID: \"0a8e24b7-7139-43b6-90b9-20860369b7bf\") "
Jul 10 19:20:55.442187 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.442167    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a8e24b7-7139-43b6-90b9-20860369b7bf" (UID: "0a8e24b7-7139-43b6-90b9-20860369b7bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:20:55.443829 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.443803    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst" (OuterVolumeSpecName: "kube-api-access-gnbst") pod "0a8e24b7-7139-43b6-90b9-20860369b7bf" (UID: "0a8e24b7-7139-43b6-90b9-20860369b7bf"). InnerVolumeSpecName "kube-api-access-gnbst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:20:55.443945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.443876    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a8e24b7-7139-43b6-90b9-20860369b7bf" (UID: "0a8e24b7-7139-43b6-90b9-20860369b7bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:20:55.491457 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.491398    2550 generic.go:356] "Generic (PLEG): container finished" podID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerID="6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c" exitCode=0
Jul 10 19:20:55.491457 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.491445    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"
Jul 10 19:20:55.491881 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.491481    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerDied","Data":"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"}
Jul 10 19:20:55.491881 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.491542    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4" event={"ID":"0a8e24b7-7139-43b6-90b9-20860369b7bf","Type":"ContainerDied","Data":"d4d3131a9a0a71f7dfa6463057aa0a93cd7bd673e6cc0631e0ca75a402dba2ed"}
Jul 10 19:20:55.491881 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.491580    2550 scope.go:117] "RemoveContainer" containerID="6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"
Jul 10 19:20:55.493271 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.493252    2550 generic.go:356] "Generic (PLEG): container finished" podID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerID="a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960" exitCode=0
Jul 10 19:20:55.493355 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.493334    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerDied","Data":"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"}
Jul 10 19:20:55.493423 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.493408    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2" event={"ID":"4973cc2d-3c76-450c-887f-dc31c7e95859","Type":"ContainerDied","Data":"72cab0bdfd727c969bf15ce3b35ae6810928a6af4c4e52cb6a07fa4b76025e1f"}
Jul 10 19:20:55.493550 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.493532    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"
Jul 10 19:20:55.493673 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.493654    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:20:55.501080 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.499999    2550 scope.go:117] "RemoveContainer" containerID="6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"
Jul 10 19:20:55.510191 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.510169    2550 scope.go:117] "RemoveContainer" containerID="6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"
Jul 10 19:20:55.510464 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:55.510444    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa\": container with ID starting with 6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa not found: ID does not exist" containerID="6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"
Jul 10 19:20:55.510552 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.510469    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa"} err="failed to get container status \"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa\": rpc error: code = NotFound desc = could not find container \"6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa\": container with ID starting with 6b556f3a662ffb7a589aed4ca4180a81de9622c54ced56267fa613bb840657fa not found: ID does not exist"
Jul 10 19:20:55.510552 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.510484    2550 scope.go:117] "RemoveContainer" containerID="6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"
Jul 10 19:20:55.510765 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:55.510719    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c\": container with ID starting with 6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c not found: ID does not exist" containerID="6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"
Jul 10 19:20:55.510837 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.510773    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c"} err="failed to get container status \"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c\": rpc error: code = NotFound desc = could not find container \"6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c\": container with ID starting with 6e2de343b06eca48224ace96ba0cb81d70a00436ca4276289467dde77875ce2c not found: ID does not exist"
Jul 10 19:20:55.510837 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.510789    2550 scope.go:117] "RemoveContainer" containerID="649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"
Jul 10 19:20:55.518179 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.518163    2550 scope.go:117] "RemoveContainer" containerID="a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"
Jul 10 19:20:55.518335 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.518286    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:20:55.522623 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.522602    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3dba1-predictor-6f64fbb6b5-cdmf4"]
Jul 10 19:20:55.525604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.525590    2550 scope.go:117] "RemoveContainer" containerID="649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"
Jul 10 19:20:55.525898 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:55.525875    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f\": container with ID starting with 649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f not found: ID does not exist" containerID="649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"
Jul 10 19:20:55.525970 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.525907    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f"} err="failed to get container status \"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f\": rpc error: code = NotFound desc = could not find container \"649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f\": container with ID starting with 649a32a791a5d2aac3f189a62070394e33ec7a56b015f4c608b62c9b3b3bff5f not found: ID does not exist"
Jul 10 19:20:55.525970 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.525932    2550 scope.go:117] "RemoveContainer" containerID="a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"
Jul 10 19:20:55.526225 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:20:55.526208    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960\": container with ID starting with a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960 not found: ID does not exist" containerID="a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"
Jul 10 19:20:55.526263 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.526231    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960"} err="failed to get container status \"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960\": rpc error: code = NotFound desc = could not find container \"a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960\": container with ID starting with a18b8ce3f0d5d6e603631b2e4e748125c07db8eef27b3af0c9a734b5819e5960 not found: ID does not exist"
Jul 10 19:20:55.532882 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.532864    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:20:55.535157 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.535138    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3dba1-predictor-6ccc87694d-zk5s2"]
Jul 10 19:20:55.543031 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.543013    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbst\" (UniqueName: \"kubernetes.io/projected/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-api-access-gnbst\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:55.543096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.543038    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a8e24b7-7139-43b6-90b9-20860369b7bf-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:55.543096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:55.543054    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a8e24b7-7139-43b6-90b9-20860369b7bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:20:56.274963 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:56.274884    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" path="/var/lib/kubelet/pods/0a8e24b7-7139-43b6-90b9-20860369b7bf/volumes"
Jul 10 19:20:56.275271 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:56.275259    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" path="/var/lib/kubelet/pods/4973cc2d-3c76-450c-887f-dc31c7e95859/volumes"
Jul 10 19:20:56.339165 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:56.339125    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 10 19:20:58.485754 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:58.485717    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:20:58.486216 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:20:58.486190    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:21:00.499081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:00.499051    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:21:00.499662 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:00.499636    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:21:05.332454 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:05.332417    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:21:06.340423 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:06.340398    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:21:08.486831 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:08.486797    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:21:10.500409 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:10.500364    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:21:18.486233 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:18.486196    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:21:20.499865 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:20.499824    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:21:27.617935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.617896    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:21:27.618336 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.618136    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container" containerID="cri-o://1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac" gracePeriod=30
Jul 10 19:21:27.618336 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.618200    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kube-rbac-proxy" containerID="cri-o://89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2" gracePeriod=30
Jul 10 19:21:27.671981 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.671948    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:21:27.672268 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.672240    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container" containerID="cri-o://d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b" gracePeriod=30
Jul 10 19:21:27.672469 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.672405    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kube-rbac-proxy" containerID="cri-o://99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda" gracePeriod=30
Jul 10 19:21:27.693434 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693405    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:21:27.693794 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693777    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693796    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693822    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693831    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693840    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kube-rbac-proxy"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693848    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kube-rbac-proxy"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693863    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kube-rbac-proxy"
Jul 10 19:21:27.693884 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693871    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kube-rbac-proxy"
Jul 10 19:21:27.694218 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693953    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kserve-container"
Jul 10 19:21:27.694218 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693967    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kube-rbac-proxy"
Jul 10 19:21:27.694218 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693977    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8e24b7-7139-43b6-90b9-20860369b7bf" containerName="kube-rbac-proxy"
Jul 10 19:21:27.694218 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.693986    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4973cc2d-3c76-450c-887f-dc31c7e95859" containerName="kserve-container"
Jul 10 19:21:27.698287 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.698267    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.700905 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.700883    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e7fe-kube-rbac-proxy-sar-config\""
Jul 10 19:21:27.701010 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.700900    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e7fe-predictor-serving-cert\""
Jul 10 19:21:27.705320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.705221    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:21:27.787769 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.787729    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:21:27.791042 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.791022    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:27.793441 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.793420    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5e7fe-predictor-serving-cert\""
Jul 10 19:21:27.793534 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.793420    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5e7fe-kube-rbac-proxy-sar-config\""
Jul 10 19:21:27.800203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.800109    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:21:27.809453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.809288    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.809453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.809361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj2c\" (UniqueName: \"kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.809453 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.809430    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.910461 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910430    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.910643 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910488    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:27.910643 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:27.910643 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910610    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.910643 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910638    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w26c\" (UniqueName: \"kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:27.910864 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.910701    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj2c\" (UniqueName: \"kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.910864 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:27.910716    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-serving-cert: secret "success-200-isvc-5e7fe-predictor-serving-cert" not found
Jul 10 19:21:27.910864 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:27.910842    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls podName:7565991f-b712-459f-8763-d4f09111de7f nodeName:}" failed. No retries permitted until 2026-07-10 19:21:28.410820554 +0000 UTC m=+1374.709171902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls") pod "success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" (UID: "7565991f-b712-459f-8763-d4f09111de7f") : secret "success-200-isvc-5e7fe-predictor-serving-cert" not found
Jul 10 19:21:27.911210 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.911192    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:27.921083 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:27.921058    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj2c\" (UniqueName: \"kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:28.011350 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.011312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.011502 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.011365    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w26c\" (UniqueName: \"kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.011502 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.011421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.011590 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:28.011557    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-serving-cert: secret "error-404-isvc-5e7fe-predictor-serving-cert" not found
Jul 10 19:21:28.011631 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:28.011611    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls podName:03d4b1b6-53bd-44bd-822e-782b747038f4 nodeName:}" failed. No retries permitted until 2026-07-10 19:21:28.51159381 +0000 UTC m=+1374.809945181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls") pod "error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" (UID: "03d4b1b6-53bd-44bd-822e-782b747038f4") : secret "error-404-isvc-5e7fe-predictor-serving-cert" not found
Jul 10 19:21:28.012061 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.012042    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.022039 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.022015    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w26c\" (UniqueName: \"kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.416163 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.416125    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:28.418662 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.418638    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") pod \"success-200-isvc-5e7fe-predictor-68b56c976c-rx55x\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:28.486257 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.486221    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:21:28.517342 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.517307    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.519747 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.519714    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") pod \"error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") " pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.611155 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.611106    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:28.631787 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.631759    2550 generic.go:356] "Generic (PLEG): container finished" podID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerID="89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2" exitCode=2
Jul 10 19:21:28.632224 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.631851    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerDied","Data":"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"}
Jul 10 19:21:28.634223 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.634198    2550 generic.go:356] "Generic (PLEG): container finished" podID="17b39e35-525f-4039-88b4-50169241e202" containerID="99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda" exitCode=2
Jul 10 19:21:28.634334 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.634237    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerDied","Data":"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"}
Jul 10 19:21:28.701797 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.701512    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:28.751087 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.750632    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:21:28.754241 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:21:28.754210    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7565991f_b712_459f_8763_d4f09111de7f.slice/crio-bff279e8eb6d566cc5a7cb5e7eeef9e872b2d7d37a239df420abbabceb79d1d7 WatchSource:0}: Error finding container bff279e8eb6d566cc5a7cb5e7eeef9e872b2d7d37a239df420abbabceb79d1d7: Status 404 returned error can't find the container with id bff279e8eb6d566cc5a7cb5e7eeef9e872b2d7d37a239df420abbabceb79d1d7
Jul 10 19:21:28.852647 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:28.852627    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:21:28.862901 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:21:28.862877    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d4b1b6_53bd_44bd_822e_782b747038f4.slice/crio-62bbdc4ed673d3a58c04ec731475dfd93b0a678eac0b092cbf9e00b9745691c9 WatchSource:0}: Error finding container 62bbdc4ed673d3a58c04ec731475dfd93b0a678eac0b092cbf9e00b9745691c9: Status 404 returned error can't find the container with id 62bbdc4ed673d3a58c04ec731475dfd93b0a678eac0b092cbf9e00b9745691c9
Jul 10 19:21:29.640616 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.640581    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerStarted","Data":"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"}
Jul 10 19:21:29.641090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.640685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerStarted","Data":"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"}
Jul 10 19:21:29.641090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.640710    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerStarted","Data":"62bbdc4ed673d3a58c04ec731475dfd93b0a678eac0b092cbf9e00b9745691c9"}
Jul 10 19:21:29.641090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.640757    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:29.641090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.640884    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:29.642523 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.642495    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:21:29.642692 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.642668    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerStarted","Data":"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"}
Jul 10 19:21:29.642785 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.642759    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerStarted","Data":"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"}
Jul 10 19:21:29.642827 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.642796    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerStarted","Data":"bff279e8eb6d566cc5a7cb5e7eeef9e872b2d7d37a239df420abbabceb79d1d7"}
Jul 10 19:21:29.642864 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.642831    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:29.658150 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.658103    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podStartSLOduration=2.658089259 podStartE2EDuration="2.658089259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:21:29.657577067 +0000 UTC m=+1375.955928438" watchObservedRunningTime="2026-07-10 19:21:29.658089259 +0000 UTC m=+1375.956440627"
Jul 10 19:21:29.675740 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:29.675695    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podStartSLOduration=2.675681503 podStartE2EDuration="2.675681503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:21:29.674513196 +0000 UTC m=+1375.972864565" watchObservedRunningTime="2026-07-10 19:21:29.675681503 +0000 UTC m=+1375.974032872"
Jul 10 19:21:30.326794 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:30.326695    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 10 19:21:30.500236 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:30.500195    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 10 19:21:30.647392 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:30.647354    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:21:30.647873 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:30.647561    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:30.648811 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:30.648785    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:21:31.026470 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.026445    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:21:31.138724 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.138690    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") pod \"17b39e35-525f-4039-88b4-50169241e202\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") "
Jul 10 19:21:31.138935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.138763    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config\") pod \"17b39e35-525f-4039-88b4-50169241e202\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") "
Jul 10 19:21:31.138935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.138797    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnfgz\" (UniqueName: \"kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz\") pod \"17b39e35-525f-4039-88b4-50169241e202\" (UID: \"17b39e35-525f-4039-88b4-50169241e202\") "
Jul 10 19:21:31.139162 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.139135    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17b39e35-525f-4039-88b4-50169241e202" (UID: "17b39e35-525f-4039-88b4-50169241e202"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:31.140930 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.140902    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17b39e35-525f-4039-88b4-50169241e202" (UID: "17b39e35-525f-4039-88b4-50169241e202"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:31.140930 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.140925    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz" (OuterVolumeSpecName: "kube-api-access-bnfgz") pod "17b39e35-525f-4039-88b4-50169241e202" (UID: "17b39e35-525f-4039-88b4-50169241e202"). InnerVolumeSpecName "kube-api-access-bnfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:31.240146 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.240066    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b39e35-525f-4039-88b4-50169241e202-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:31.240146 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.240095    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b39e35-525f-4039-88b4-50169241e202-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:31.240146 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.240104    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfgz\" (UniqueName: \"kubernetes.io/projected/17b39e35-525f-4039-88b4-50169241e202-kube-api-access-bnfgz\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:31.465677 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.465645    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:21:31.644060 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.644032    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhcc9\" (UniqueName: \"kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9\") pod \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") "
Jul 10 19:21:31.644244 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.644103    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls\") pod \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") "
Jul 10 19:21:31.644244 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.644146    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config\") pod \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\" (UID: \"fc4d78ba-478a-41a2-bb7c-77744b61bb2f\") "
Jul 10 19:21:31.644559 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.644533    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc4d78ba-478a-41a2-bb7c-77744b61bb2f" (UID: "fc4d78ba-478a-41a2-bb7c-77744b61bb2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:31.646199 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.646161    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9" (OuterVolumeSpecName: "kube-api-access-dhcc9") pod "fc4d78ba-478a-41a2-bb7c-77744b61bb2f" (UID: "fc4d78ba-478a-41a2-bb7c-77744b61bb2f"). InnerVolumeSpecName "kube-api-access-dhcc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:31.646299 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.646265    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc4d78ba-478a-41a2-bb7c-77744b61bb2f" (UID: "fc4d78ba-478a-41a2-bb7c-77744b61bb2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:31.652636 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.652611    2550 generic.go:356] "Generic (PLEG): container finished" podID="17b39e35-525f-4039-88b4-50169241e202" containerID="d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b" exitCode=0
Jul 10 19:21:31.653008 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.652672    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"
Jul 10 19:21:31.653008 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.652691    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerDied","Data":"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"}
Jul 10 19:21:31.653008 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.652798    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt" event={"ID":"17b39e35-525f-4039-88b4-50169241e202","Type":"ContainerDied","Data":"7191b17bf466665eb3bce77d8dfbc15a3a897bd1e0dbeae26741f31b26d22bee"}
Jul 10 19:21:31.653008 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.652841    2550 scope.go:117] "RemoveContainer" containerID="99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"
Jul 10 19:21:31.654440 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.654418    2550 generic.go:356] "Generic (PLEG): container finished" podID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerID="1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac" exitCode=0
Jul 10 19:21:31.654528 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.654475    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"
Jul 10 19:21:31.654528 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.654484    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerDied","Data":"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"}
Jul 10 19:21:31.654645 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.654542    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8" event={"ID":"fc4d78ba-478a-41a2-bb7c-77744b61bb2f","Type":"ContainerDied","Data":"956cee25a4b36921300bcdf9d675afd2250094ee2050bac5ebf866286a025709"}
Jul 10 19:21:31.654940 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.654900    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:21:31.662151 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.662129    2550 scope.go:117] "RemoveContainer" containerID="d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"
Jul 10 19:21:31.669634 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.669618    2550 scope.go:117] "RemoveContainer" containerID="99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"
Jul 10 19:21:31.669910 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:31.669891    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda\": container with ID starting with 99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda not found: ID does not exist" containerID="99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"
Jul 10 19:21:31.669968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.669919    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda"} err="failed to get container status \"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda\": rpc error: code = NotFound desc = could not find container \"99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda\": container with ID starting with 99b448f94784b3c3367fceba32d4efd28280f084fb39b9163103c23d9daaaeda not found: ID does not exist"
Jul 10 19:21:31.669968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.669935    2550 scope.go:117] "RemoveContainer" containerID="d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"
Jul 10 19:21:31.670198 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:31.670181    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b\": container with ID starting with d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b not found: ID does not exist" containerID="d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"
Jul 10 19:21:31.670285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.670202    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b"} err="failed to get container status \"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b\": rpc error: code = NotFound desc = could not find container \"d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b\": container with ID starting with d1bf1963d7c2adaa6eb63cf67c1686ce585b67d4f9bf08f99601bee34221153b not found: ID does not exist"
Jul 10 19:21:31.670285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.670217    2550 scope.go:117] "RemoveContainer" containerID="89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"
Jul 10 19:21:31.677331 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.677314    2550 scope.go:117] "RemoveContainer" containerID="1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"
Jul 10 19:21:31.678044 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.678027    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:21:31.682090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.682072    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d30ee-predictor-b984bbfb8-jpqd8"]
Jul 10 19:21:31.684779 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.684759    2550 scope.go:117] "RemoveContainer" containerID="89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"
Jul 10 19:21:31.685084 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:31.685061    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2\": container with ID starting with 89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2 not found: ID does not exist" containerID="89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"
Jul 10 19:21:31.685197 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.685093    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2"} err="failed to get container status \"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2\": rpc error: code = NotFound desc = could not find container \"89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2\": container with ID starting with 89414c0c2f731f3737ac2424ee3110becfb67fb57a8e18949b99a6209d58efa2 not found: ID does not exist"
Jul 10 19:21:31.685197 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.685118    2550 scope.go:117] "RemoveContainer" containerID="1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"
Jul 10 19:21:31.685382 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:21:31.685366    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac\": container with ID starting with 1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac not found: ID does not exist" containerID="1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"
Jul 10 19:21:31.685426 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.685386    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac"} err="failed to get container status \"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac\": rpc error: code = NotFound desc = could not find container \"1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac\": container with ID starting with 1c86be6e6be2f1ed993c7f150ed20ffcca7be0a91bfdd61b76a886657dd413ac not found: ID does not exist"
Jul 10 19:21:31.692802 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.692784    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:21:31.696635 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.696613    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30ee-predictor-585b9dc857-dqrvt"]
Jul 10 19:21:31.744696 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.744662    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:31.744696 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.744695    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:31.744938 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:31.744716    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcc9\" (UniqueName: \"kubernetes.io/projected/fc4d78ba-478a-41a2-bb7c-77744b61bb2f-kube-api-access-dhcc9\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.274626 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:32.274595    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b39e35-525f-4039-88b4-50169241e202" path="/var/lib/kubelet/pods/17b39e35-525f-4039-88b4-50169241e202/volumes"
Jul 10 19:21:32.275018 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:32.275001    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" path="/var/lib/kubelet/pods/fc4d78ba-478a-41a2-bb7c-77744b61bb2f/volumes"
Jul 10 19:21:35.652329 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:35.652296    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:21:35.652859 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:35.652833    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:21:36.659639 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:36.659609    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:21:36.660077 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:36.660051    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:21:38.486317 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:38.486283    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:21:40.501085 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:40.501054    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:21:45.653180 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:45.653142    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:21:46.660855 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:46.660817    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:21:48.486815 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:48.486784    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:21:55.653692 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:55.653651    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:21:56.660683 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:21:56.660642    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:22:05.653031 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:22:05.652993    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:22:06.660683 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:22:06.660642    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 10 19:22:15.653914 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:22:15.653878    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:22:16.664131 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:22:16.664097    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:30:16.438422 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.438347    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:30:16.438971 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.438553    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" containerID="cri-o://ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1" gracePeriod=30
Jul 10 19:30:16.438971 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.438621    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kube-rbac-proxy" containerID="cri-o://8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc" gracePeriod=30
Jul 10 19:30:16.499283 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499252    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:30:16.499721 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499697    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container"
Jul 10 19:30:16.499721 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499722    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499776    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kube-rbac-proxy"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499786    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kube-rbac-proxy"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499815    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499823    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499834    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kube-rbac-proxy"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499841    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kube-rbac-proxy"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499927    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kserve-container"
Jul 10 19:30:16.499939 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499943    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kube-rbac-proxy"
Jul 10 19:30:16.500325 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499954    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4d78ba-478a-41a2-bb7c-77744b61bb2f" containerName="kserve-container"
Jul 10 19:30:16.500325 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.499964    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b39e35-525f-4039-88b4-50169241e202" containerName="kube-rbac-proxy"
Jul 10 19:30:16.503292 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.503268    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.505823 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.505799    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b29b-predictor-serving-cert\""
Jul 10 19:30:16.505936 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.505803    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b29b-kube-rbac-proxy-sar-config\""
Jul 10 19:30:16.512796 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.512459    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:30:16.557802 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.557764    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:30:16.560950 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.560931    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.563483 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.563462    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b29b-kube-rbac-proxy-sar-config\""
Jul 10 19:30:16.563585 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.563507    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b29b-predictor-serving-cert\""
Jul 10 19:30:16.571528 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.571508    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:30:16.609224 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.609195    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:30:16.609555 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.609497    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container" containerID="cri-o://9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889" gracePeriod=30
Jul 10 19:30:16.609555 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.609524    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kube-rbac-proxy" containerID="cri-o://fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a" gracePeriod=30
Jul 10 19:30:16.655401 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.655369    2550 generic.go:356] "Generic (PLEG): container finished" podID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerID="8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc" exitCode=2
Jul 10 19:30:16.655516 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.655406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerDied","Data":"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"}
Jul 10 19:30:16.679832 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679809    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.679945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679850    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.679945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jt8\" (UniqueName: \"kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.679945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58652\" (UniqueName: \"kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.679945 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679914    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.680167 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.679966    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.781096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781071    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.781228 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781110    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.781228 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781129    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jt8\" (UniqueName: \"kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.781228 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781166    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58652\" (UniqueName: \"kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.781381 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.781434 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781396    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.781546 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:16.781528    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-serving-cert: secret "error-404-isvc-4b29b-predictor-serving-cert" not found
Jul 10 19:30:16.781640 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:16.781611    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls podName:9e8dabdf-ef51-425b-a17d-cfacffb419d3 nodeName:}" failed. No retries permitted until 2026-07-10 19:30:17.281590761 +0000 UTC m=+1903.579942108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls") pod "error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" (UID: "9e8dabdf-ef51-425b-a17d-cfacffb419d3") : secret "error-404-isvc-4b29b-predictor-serving-cert" not found
Jul 10 19:30:16.781979 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.781959    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.782093 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.782047    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.783538 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.783516    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.790181 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.790138    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jt8\" (UniqueName: \"kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:16.790494 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.790465    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58652\" (UniqueName: \"kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652\") pod \"success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") " pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.816314 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.816292    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:16.948285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:16.948195    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:30:16.950943 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:30:16.950915    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadf3a41_21b5_4f85_8bad_65cce3ff530a.slice/crio-b475ee8021db4ff9a56e6705741460265db0a34c9467b234b33b87c640f38220 WatchSource:0}: Error finding container b475ee8021db4ff9a56e6705741460265db0a34c9467b234b33b87c640f38220: Status 404 returned error can't find the container with id b475ee8021db4ff9a56e6705741460265db0a34c9467b234b33b87c640f38220
Jul 10 19:30:17.285525 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.285430    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:17.287900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.287883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") pod \"error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") " pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:17.471530 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.471490    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:17.603986 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.603946    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:30:17.607495 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:30:17.607450    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8dabdf_ef51_425b_a17d_cfacffb419d3.slice/crio-e97d2fe0ab1745fd728e3d8d5a14f6dcddaa2f00f7326042645ebc33926ae46d WatchSource:0}: Error finding container e97d2fe0ab1745fd728e3d8d5a14f6dcddaa2f00f7326042645ebc33926ae46d: Status 404 returned error can't find the container with id e97d2fe0ab1745fd728e3d8d5a14f6dcddaa2f00f7326042645ebc33926ae46d
Jul 10 19:30:17.660795 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.660760    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerStarted","Data":"e97d2fe0ab1745fd728e3d8d5a14f6dcddaa2f00f7326042645ebc33926ae46d"}
Jul 10 19:30:17.663016 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.662994    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerStarted","Data":"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"}
Jul 10 19:30:17.663118 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.663046    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerStarted","Data":"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"}
Jul 10 19:30:17.663118 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.663073    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerStarted","Data":"b475ee8021db4ff9a56e6705741460265db0a34c9467b234b33b87c640f38220"}
Jul 10 19:30:17.663227 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.663163    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:17.664777 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.664755    2550 generic.go:356] "Generic (PLEG): container finished" podID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerID="fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a" exitCode=2
Jul 10 19:30:17.664877 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.664834    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerDied","Data":"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"}
Jul 10 19:30:17.681418 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:17.681376    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podStartSLOduration=1.681360042 podStartE2EDuration="1.681360042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:17.679125012 +0000 UTC m=+1903.977476382" watchObservedRunningTime="2026-07-10 19:30:17.681360042 +0000 UTC m=+1903.979711411"
Jul 10 19:30:18.483045 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.483009    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 10 19:30:18.486237 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.486212    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 10 19:30:18.670916 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.670882    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerStarted","Data":"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"}
Jul 10 19:30:18.671110 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.671061    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:18.671184 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.671158    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:18.671241 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.671194    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerStarted","Data":"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"}
Jul 10 19:30:18.671367 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.671350    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:18.672195 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.672168    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:18.672331 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.672177    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:18.694482 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:18.694440    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podStartSLOduration=2.694428392 podStartE2EDuration="2.694428392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:18.692485496 +0000 UTC m=+1904.990836888" watchObservedRunningTime="2026-07-10 19:30:18.694428392 +0000 UTC m=+1904.992779759"
Jul 10 19:30:19.594563 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.594536    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:30:19.676299 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676214    2550 generic.go:356] "Generic (PLEG): container finished" podID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerID="ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1" exitCode=0
Jul 10 19:30:19.676299 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676273    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"
Jul 10 19:30:19.676299 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676275    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerDied","Data":"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"}
Jul 10 19:30:19.676515 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c" event={"ID":"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c","Type":"ContainerDied","Data":"68925ac1326cd914b0c1f669cec0c9257f8e7efe5e75dd9e27c9f8562158a4d3"}
Jul 10 19:30:19.676515 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676418    2550 scope.go:117] "RemoveContainer" containerID="8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"
Jul 10 19:30:19.677021 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.676989    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:19.677180 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.677144    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:19.684668 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.684652    2550 scope.go:117] "RemoveContainer" containerID="ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"
Jul 10 19:30:19.691869 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.691841    2550 scope.go:117] "RemoveContainer" containerID="8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"
Jul 10 19:30:19.692101 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:19.692084    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc\": container with ID starting with 8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc not found: ID does not exist" containerID="8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"
Jul 10 19:30:19.692160 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.692112    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc"} err="failed to get container status \"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc\": rpc error: code = NotFound desc = could not find container \"8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc\": container with ID starting with 8f3cc0db90e4ae72c5efdb828fc0cbb1b82027dd61d0491a8ed8cae70ca25ffc not found: ID does not exist"
Jul 10 19:30:19.692160 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.692129    2550 scope.go:117] "RemoveContainer" containerID="ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"
Jul 10 19:30:19.692384 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:19.692367    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1\": container with ID starting with ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1 not found: ID does not exist" containerID="ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"
Jul 10 19:30:19.692447 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.692394    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1"} err="failed to get container status \"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1\": rpc error: code = NotFound desc = could not find container \"ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1\": container with ID starting with ab668e1b019a03f7fb06b67fa4f06d899a1e05b79e5b65ff3528348017dde6b1 not found: ID does not exist"
Jul 10 19:30:19.706123 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.706105    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqz4h\" (UniqueName: \"kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h\") pod \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") "
Jul 10 19:30:19.706229 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.706144    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config\") pod \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") "
Jul 10 19:30:19.706229 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.706178    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls\") pod \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\" (UID: \"08e9d3ba-f634-4ed4-b81e-d82a66f50c3c\") "
Jul 10 19:30:19.706542 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.706514    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" (UID: "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:30:19.708304 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.708272    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" (UID: "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:30:19.708475 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.708452    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h" (OuterVolumeSpecName: "kube-api-access-fqz4h") pod "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" (UID: "08e9d3ba-f634-4ed4-b81e-d82a66f50c3c"). InnerVolumeSpecName "kube-api-access-fqz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:30:19.806983 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.806958    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:19.806983 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.806985    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:19.807179 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.807000    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqz4h\" (UniqueName: \"kubernetes.io/projected/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c-kube-api-access-fqz4h\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:19.947333 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.947303    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:30:19.998727 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:19.998696    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:30:20.000703 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.000672    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d232d-predictor-5459ccd6b9-ldx4c"]
Jul 10 19:30:20.009546 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.009521    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2lgr\" (UniqueName: \"kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr\") pod \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") "
Jul 10 19:30:20.009662 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.009563    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") pod \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") "
Jul 10 19:30:20.011771 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.011720    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" (UID: "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:30:20.011868 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.011769    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr" (OuterVolumeSpecName: "kube-api-access-q2lgr") pod "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" (UID: "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0"). InnerVolumeSpecName "kube-api-access-q2lgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:30:20.110064 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.110026    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config\") pod \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\" (UID: \"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0\") "
Jul 10 19:30:20.110253 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.110207    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2lgr\" (UniqueName: \"kubernetes.io/projected/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-api-access-q2lgr\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:20.110253 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.110225    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:20.110380 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.110355    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" (UID: "1d8b4dd5-a9be-4ded-8f51-c3f498f541b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:30:20.211366 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.211290    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:20.275061 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.275036    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" path="/var/lib/kubelet/pods/08e9d3ba-f634-4ed4-b81e-d82a66f50c3c/volumes"
Jul 10 19:30:20.683129 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683100    2550 generic.go:356] "Generic (PLEG): container finished" podID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerID="9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889" exitCode=0
Jul 10 19:30:20.683560 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683153    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"
Jul 10 19:30:20.683560 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683185    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerDied","Data":"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"}
Jul 10 19:30:20.683560 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683243    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh" event={"ID":"1d8b4dd5-a9be-4ded-8f51-c3f498f541b0","Type":"ContainerDied","Data":"d264eb207ac47fa31e4ad43de850df615814019606a337de982730e26edd6f72"}
Jul 10 19:30:20.683560 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683271    2550 scope.go:117] "RemoveContainer" containerID="fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"
Jul 10 19:30:20.683830 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.683803    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:20.691402 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.691385    2550 scope.go:117] "RemoveContainer" containerID="9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"
Jul 10 19:30:20.698635 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.698617    2550 scope.go:117] "RemoveContainer" containerID="fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"
Jul 10 19:30:20.698919 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:20.698899    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a\": container with ID starting with fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a not found: ID does not exist" containerID="fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"
Jul 10 19:30:20.698981 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.698926    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a"} err="failed to get container status \"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a\": rpc error: code = NotFound desc = could not find container \"fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a\": container with ID starting with fe35dfef3589a84756462e7818430e45b20990992a0c85a25d6aac5dcd85cd9a not found: ID does not exist"
Jul 10 19:30:20.698981 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.698942    2550 scope.go:117] "RemoveContainer" containerID="9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"
Jul 10 19:30:20.699218 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:20.699193    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889\": container with ID starting with 9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889 not found: ID does not exist" containerID="9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"
Jul 10 19:30:20.699268 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.699227    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889"} err="failed to get container status \"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889\": rpc error: code = NotFound desc = could not find container \"9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889\": container with ID starting with 9d2267f56bc60ca836e8c1f865a17e7a5630a6599dd9f20503717bf6cc977889 not found: ID does not exist"
Jul 10 19:30:20.701265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.701246    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:30:20.705383 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:20.705365    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d232d-predictor-6d6f664f96-bzwzh"]
Jul 10 19:30:22.274497 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:22.274455    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" path="/var/lib/kubelet/pods/1d8b4dd5-a9be-4ded-8f51-c3f498f541b0/volumes"
Jul 10 19:30:24.681868 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:24.681837    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:30:24.682432 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:24.682404    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:25.688223 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:25.688196    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:30:25.688855 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:25.688820    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:34.682938 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:34.682895    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:35.689666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:35.689627    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:42.478999 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.478971    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:30:42.479404 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.479170    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container" containerID="cri-o://088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620" gracePeriod=30
Jul 10 19:30:42.479404 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.479213    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kube-rbac-proxy" containerID="cri-o://765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3" gracePeriod=30
Jul 10 19:30:42.542644 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.542610    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:30:42.542862 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.542842    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" containerID="cri-o://00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08" gracePeriod=30
Jul 10 19:30:42.542946 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.542893    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kube-rbac-proxy" containerID="cri-o://f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7" gracePeriod=30
Jul 10 19:30:42.574841 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.574814    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:30:42.575150 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575138    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kube-rbac-proxy"
Jul 10 19:30:42.575190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575152    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kube-rbac-proxy"
Jul 10 19:30:42.575190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575161    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kube-rbac-proxy"
Jul 10 19:30:42.575190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575166    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kube-rbac-proxy"
Jul 10 19:30:42.575190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575180    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container"
Jul 10 19:30:42.575190 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575185    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575192    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575197    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575242    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kserve-container"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575250    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e9d3ba-f634-4ed4-b81e-d82a66f50c3c" containerName="kube-rbac-proxy"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575255    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kserve-container"
Jul 10 19:30:42.575340 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.575262    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8b4dd5-a9be-4ded-8f51-c3f498f541b0" containerName="kube-rbac-proxy"
Jul 10 19:30:42.578062 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.578046    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.580609 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.580587    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1c787-predictor-serving-cert\""
Jul 10 19:30:42.580717 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.580587    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1c787-kube-rbac-proxy-sar-config\""
Jul 10 19:30:42.591697 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.591673    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:30:42.667458 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.667429    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msc4f\" (UniqueName: \"kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.667603 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.667500    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.667603 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.667554    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.683149 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.683129    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:30:42.686361 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.686340    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.689812 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.689791    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1c787-kube-rbac-proxy-sar-config\""
Jul 10 19:30:42.690351 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.690331    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1c787-predictor-serving-cert\""
Jul 10 19:30:42.697501 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.697478    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:30:42.768472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768389    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msc4f\" (UniqueName: \"kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.768472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768431    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.768472 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768458    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbw2m\" (UniqueName: \"kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.768776 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768492    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.768776 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768514    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.768776 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.768541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.769244 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.769219    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.771207 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.771184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.774921 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.774899    2550 generic.go:356] "Generic (PLEG): container finished" podID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerID="f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7" exitCode=2
Jul 10 19:30:42.775017 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.774967    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerDied","Data":"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"}
Jul 10 19:30:42.777153 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.777121    2550 generic.go:356] "Generic (PLEG): container finished" podID="7565991f-b712-459f-8763-d4f09111de7f" containerID="765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3" exitCode=2
Jul 10 19:30:42.777259 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.777213    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerDied","Data":"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"}
Jul 10 19:30:42.777997 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.777977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msc4f\" (UniqueName: \"kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f\") pod \"success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") " pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:42.869756 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.869706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.869900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.869791    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.869900 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.869820    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbw2m\" (UniqueName: \"kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.869900 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:42.869857    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1c787-predictor-serving-cert: secret "error-404-isvc-1c787-predictor-serving-cert" not found
Jul 10 19:30:42.870006 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:42.869921    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls podName:d43689cc-70b7-41fd-9e17-a1f9971ae6ee nodeName:}" failed. No retries permitted until 2026-07-10 19:30:43.369904352 +0000 UTC m=+1929.668255699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls") pod "error-404-isvc-1c787-predictor-794854c758-r7j46" (UID: "d43689cc-70b7-41fd-9e17-a1f9971ae6ee") : secret "error-404-isvc-1c787-predictor-serving-cert" not found
Jul 10 19:30:42.870539 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.870516    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.878606 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.878574    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbw2m\" (UniqueName: \"kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:42.889285 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:42.889263    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:43.034003 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.033829    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:30:43.036545 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:30:43.036516    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5992cd9f_e127_4fe4_80d1_6257a5b4e44c.slice/crio-715e9ab1ff3bfb959acf086cfc1fb65322607bebed76fe713b7870be2e02ab13 WatchSource:0}: Error finding container 715e9ab1ff3bfb959acf086cfc1fb65322607bebed76fe713b7870be2e02ab13: Status 404 returned error can't find the container with id 715e9ab1ff3bfb959acf086cfc1fb65322607bebed76fe713b7870be2e02ab13
Jul 10 19:30:43.374137 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.374055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:43.376505 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.376486    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") pod \"error-404-isvc-1c787-predictor-794854c758-r7j46\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") " pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:43.598105 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.598074    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:43.733310 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.733279    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:30:43.736159 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:30:43.736133    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43689cc_70b7_41fd_9e17_a1f9971ae6ee.slice/crio-f84ec0a5bfde94705a297476cb07fa4d97efb2aff5cb3fd014b75ff3b5c342e3 WatchSource:0}: Error finding container f84ec0a5bfde94705a297476cb07fa4d97efb2aff5cb3fd014b75ff3b5c342e3: Status 404 returned error can't find the container with id f84ec0a5bfde94705a297476cb07fa4d97efb2aff5cb3fd014b75ff3b5c342e3
Jul 10 19:30:43.782406 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.782375    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerStarted","Data":"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"}
Jul 10 19:30:43.782522 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.782443    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerStarted","Data":"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"}
Jul 10 19:30:43.782522 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.782472    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerStarted","Data":"715e9ab1ff3bfb959acf086cfc1fb65322607bebed76fe713b7870be2e02ab13"}
Jul 10 19:30:43.782522 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.782514    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:43.782684 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.782541    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:43.783516 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.783482    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerStarted","Data":"f84ec0a5bfde94705a297476cb07fa4d97efb2aff5cb3fd014b75ff3b5c342e3"}
Jul 10 19:30:43.783821 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.783710    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:30:43.809778 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:43.801539    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podStartSLOduration=1.801518992 podStartE2EDuration="1.801518992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:43.800862758 +0000 UTC m=+1930.099214131" watchObservedRunningTime="2026-07-10 19:30:43.801518992 +0000 UTC m=+1930.099870381"
Jul 10 19:30:44.682591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.682551    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:44.789202 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.789155    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerStarted","Data":"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"}
Jul 10 19:30:44.789385 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.789342    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:44.789473 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.789400    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:44.789473 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.789429    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerStarted","Data":"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"}
Jul 10 19:30:44.789591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.789556    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:30:44.790375 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.790354    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:30:44.808988 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:44.808897    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podStartSLOduration=2.808877968 podStartE2EDuration="2.808877968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:44.808028949 +0000 UTC m=+1931.106380320" watchObservedRunningTime="2026-07-10 19:30:44.808877968 +0000 UTC m=+1931.107229340"
Jul 10 19:30:45.648373 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:45.648338    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 10 19:30:45.653464 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:45.653434    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 10 19:30:45.689170 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:45.689147    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:45.793219 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:45.793176    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:30:46.355932 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.355867    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:30:46.358567 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.358549    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:30:46.395376 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395349    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") pod \"7565991f-b712-459f-8763-d4f09111de7f\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") "
Jul 10 19:30:46.395520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395396    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config\") pod \"7565991f-b712-459f-8763-d4f09111de7f\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") "
Jul 10 19:30:46.395520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395433    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zj2c\" (UniqueName: \"kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c\") pod \"7565991f-b712-459f-8763-d4f09111de7f\" (UID: \"7565991f-b712-459f-8763-d4f09111de7f\") "
Jul 10 19:30:46.395520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395464    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config\") pod \"03d4b1b6-53bd-44bd-822e-782b747038f4\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") "
Jul 10 19:30:46.395520 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395508    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") pod \"03d4b1b6-53bd-44bd-822e-782b747038f4\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") "
Jul 10 19:30:46.395717 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395535    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w26c\" (UniqueName: \"kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c\") pod \"03d4b1b6-53bd-44bd-822e-782b747038f4\" (UID: \"03d4b1b6-53bd-44bd-822e-782b747038f4\") "
Jul 10 19:30:46.395901 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395871    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7565991f-b712-459f-8763-d4f09111de7f" (UID: "7565991f-b712-459f-8763-d4f09111de7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:30:46.396008 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.395916    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03d4b1b6-53bd-44bd-822e-782b747038f4" (UID: "03d4b1b6-53bd-44bd-822e-782b747038f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:30:46.398148 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.398111    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03d4b1b6-53bd-44bd-822e-782b747038f4" (UID: "03d4b1b6-53bd-44bd-822e-782b747038f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:30:46.398235 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.398199    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7565991f-b712-459f-8763-d4f09111de7f" (UID: "7565991f-b712-459f-8763-d4f09111de7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:30:46.398319 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.398300    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c" (OuterVolumeSpecName: "kube-api-access-8w26c") pod "03d4b1b6-53bd-44bd-822e-782b747038f4" (UID: "03d4b1b6-53bd-44bd-822e-782b747038f4"). InnerVolumeSpecName "kube-api-access-8w26c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:30:46.398620 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.398596    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c" (OuterVolumeSpecName: "kube-api-access-2zj2c") pod "7565991f-b712-459f-8763-d4f09111de7f" (UID: "7565991f-b712-459f-8763-d4f09111de7f"). InnerVolumeSpecName "kube-api-access-2zj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:30:46.496691 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496656    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b1b6-53bd-44bd-822e-782b747038f4-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.496691 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496686    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w26c\" (UniqueName: \"kubernetes.io/projected/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-api-access-8w26c\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.496691 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496695    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7565991f-b712-459f-8763-d4f09111de7f-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.496960 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496703    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7565991f-b712-459f-8763-d4f09111de7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.496960 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496711    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj2c\" (UniqueName: \"kubernetes.io/projected/7565991f-b712-459f-8763-d4f09111de7f-kube-api-access-2zj2c\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.496960 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.496722    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b1b6-53bd-44bd-822e-782b747038f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:30:46.797895 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.797856    2550 generic.go:356] "Generic (PLEG): container finished" podID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerID="00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08" exitCode=0
Jul 10 19:30:46.798359 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.797913    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"
Jul 10 19:30:46.798359 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.797947    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerDied","Data":"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"}
Jul 10 19:30:46.798359 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.798015    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz" event={"ID":"03d4b1b6-53bd-44bd-822e-782b747038f4","Type":"ContainerDied","Data":"62bbdc4ed673d3a58c04ec731475dfd93b0a678eac0b092cbf9e00b9745691c9"}
Jul 10 19:30:46.798359 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.798051    2550 scope.go:117] "RemoveContainer" containerID="f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"
Jul 10 19:30:46.799625 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.799585    2550 generic.go:356] "Generic (PLEG): container finished" podID="7565991f-b712-459f-8763-d4f09111de7f" containerID="088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620" exitCode=0
Jul 10 19:30:46.799767 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.799646    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"
Jul 10 19:30:46.799767 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.799642    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerDied","Data":"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"}
Jul 10 19:30:46.800157 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.799800    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x" event={"ID":"7565991f-b712-459f-8763-d4f09111de7f","Type":"ContainerDied","Data":"bff279e8eb6d566cc5a7cb5e7eeef9e872b2d7d37a239df420abbabceb79d1d7"}
Jul 10 19:30:46.810171 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.810155    2550 scope.go:117] "RemoveContainer" containerID="00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"
Jul 10 19:30:46.817679 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.817660    2550 scope.go:117] "RemoveContainer" containerID="f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"
Jul 10 19:30:46.817937 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:46.817920    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7\": container with ID starting with f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7 not found: ID does not exist" containerID="f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"
Jul 10 19:30:46.817983 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.817946    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7"} err="failed to get container status \"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7\": rpc error: code = NotFound desc = could not find container \"f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7\": container with ID starting with f5db1e7cae3be4755f9cda3cb7b1caed92fd6ef3872707cabd157f229ca2c6d7 not found: ID does not exist"
Jul 10 19:30:46.817983 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.817964    2550 scope.go:117] "RemoveContainer" containerID="00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"
Jul 10 19:30:46.818232 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:46.818214    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08\": container with ID starting with 00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08 not found: ID does not exist" containerID="00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"
Jul 10 19:30:46.818279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.818240    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08"} err="failed to get container status \"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08\": rpc error: code = NotFound desc = could not find container \"00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08\": container with ID starting with 00381c7609c37c378eae064ac6e3c2cba64111dabc7276b79a45a29ac99e0d08 not found: ID does not exist"
Jul 10 19:30:46.818279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.818260    2550 scope.go:117] "RemoveContainer" containerID="765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"
Jul 10 19:30:46.825473 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.825456    2550 scope.go:117] "RemoveContainer" containerID="088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"
Jul 10 19:30:46.832296 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.832272    2550 scope.go:117] "RemoveContainer" containerID="765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"
Jul 10 19:30:46.832539 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:46.832517    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3\": container with ID starting with 765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3 not found: ID does not exist" containerID="765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"
Jul 10 19:30:46.832604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.832545    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3"} err="failed to get container status \"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3\": rpc error: code = NotFound desc = could not find container \"765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3\": container with ID starting with 765aee464d030d951477de51b4f53bb1e44eff73f4fae1e3e6bf6b08afa29ba3 not found: ID does not exist"
Jul 10 19:30:46.832604 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.832561    2550 scope.go:117] "RemoveContainer" containerID="088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"
Jul 10 19:30:46.832790 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:30:46.832772    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620\": container with ID starting with 088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620 not found: ID does not exist" containerID="088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"
Jul 10 19:30:46.832848 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.832803    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620"} err="failed to get container status \"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620\": rpc error: code = NotFound desc = could not find container \"088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620\": container with ID starting with 088a0268bf8c82e02ed1547f019df4648ab5505b5030184188d53c7015bae620 not found: ID does not exist"
Jul 10 19:30:46.835145 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.835126    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:30:46.842899 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.842880    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e7fe-predictor-58dbcbf874-cvkmz"]
Jul 10 19:30:46.859583 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.859562    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:30:46.865424 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:46.865394    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e7fe-predictor-68b56c976c-rx55x"]
Jul 10 19:30:48.275109 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:48.275077    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" path="/var/lib/kubelet/pods/03d4b1b6-53bd-44bd-822e-782b747038f4/volumes"
Jul 10 19:30:48.275475 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:48.275467    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7565991f-b712-459f-8763-d4f09111de7f" path="/var/lib/kubelet/pods/7565991f-b712-459f-8763-d4f09111de7f/volumes"
Jul 10 19:30:49.794336 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:49.794309    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:30:49.794791 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:49.794756    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:30:50.797517 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:50.797488    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:30:50.797958 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:50.797931    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:30:54.683274 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:54.683233    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 10 19:30:55.689584 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:55.689547    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 10 19:30:59.794968 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:30:59.794921    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:31:00.798059 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:00.798020    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:31:04.683834 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:04.683802    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:31:05.690335 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:05.690303    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:31:09.794761 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:09.794706    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:31:10.798793 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:10.798758    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:31:19.794711 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:19.794673    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 10 19:31:20.798068 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:20.798029    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:31:26.758232 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.758202    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:31:26.758728 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.758443    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container" containerID="cri-o://2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f" gracePeriod=30
Jul 10 19:31:26.758728 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.758480    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kube-rbac-proxy" containerID="cri-o://132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b" gracePeriod=30
Jul 10 19:31:26.809025 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.808988    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:31:26.809357 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.809295    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container" containerID="cri-o://17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82" gracePeriod=30
Jul 10 19:31:26.809525 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.809476    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kube-rbac-proxy" containerID="cri-o://22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b" gracePeriod=30
Jul 10 19:31:26.835238 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835209    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:31:26.835584 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835567    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kube-rbac-proxy"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835588    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kube-rbac-proxy"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835601    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kube-rbac-proxy"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835609    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kube-rbac-proxy"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835633    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835641    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835650    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container"
Jul 10 19:31:26.835666 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835658    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container"
Jul 10 19:31:26.836026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835768    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kserve-container"
Jul 10 19:31:26.836026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835781    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kube-rbac-proxy"
Jul 10 19:31:26.836026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835792    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d4b1b6-53bd-44bd-822e-782b747038f4" containerName="kserve-container"
Jul 10 19:31:26.836026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.835803    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7565991f-b712-459f-8763-d4f09111de7f" containerName="kube-rbac-proxy"
Jul 10 19:31:26.840183 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.840164    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:26.843194 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.843166    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1e1e7-predictor-serving-cert\""
Jul 10 19:31:26.843194 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.843188    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1e1e7-kube-rbac-proxy-sar-config\""
Jul 10 19:31:26.848298 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.848251    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:31:26.906265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.906233    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:26.906265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.906279    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:26.906531 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.906435    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669xm\" (UniqueName: \"kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:26.958224 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.958194    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:31:26.963206 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.962654    2550 generic.go:356] "Generic (PLEG): container finished" podID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerID="22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b" exitCode=2
Jul 10 19:31:26.963206 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.963109    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:26.963773 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.963715    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerDied","Data":"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"}
Jul 10 19:31:26.965313 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.965273    2550 generic.go:356] "Generic (PLEG): container finished" podID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerID="132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b" exitCode=2
Jul 10 19:31:26.965437 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.965339    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerDied","Data":"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"}
Jul 10 19:31:26.966261 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.966238    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e1e7-predictor-serving-cert\""
Jul 10 19:31:26.966516 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.966499    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e1e7-kube-rbac-proxy-sar-config\""
Jul 10 19:31:26.980905 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:26.980879    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:31:27.007371 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.007332    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669xm\" (UniqueName: \"kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.007542 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.007395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.007542 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.007444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.007670 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:27.007573    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-serving-cert: secret "success-200-isvc-1e1e7-predictor-serving-cert" not found
Jul 10 19:31:27.007670 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:27.007632    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls podName:4c4134c2-98cf-4853-b50d-30ac551f5683 nodeName:}" failed. No retries permitted until 2026-07-10 19:31:27.507611673 +0000 UTC m=+1973.805963026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls") pod "success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" (UID: "4c4134c2-98cf-4853-b50d-30ac551f5683") : secret "success-200-isvc-1e1e7-predictor-serving-cert" not found
Jul 10 19:31:27.008421 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.008326    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.024950 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.024928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669xm\" (UniqueName: \"kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.108186 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.108097    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h57h\" (UniqueName: \"kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.108467 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.108201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.108467 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.108255    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.209024 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.208981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.209231 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.209074    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h57h\" (UniqueName: \"kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.209231 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.209138    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.209350 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:27.209281    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-serving-cert: secret "error-404-isvc-1e1e7-predictor-serving-cert" not found
Jul 10 19:31:27.209350 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:27.209338    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls podName:b86ae9c0-0709-410b-82ee-42846b74d896 nodeName:}" failed. No retries permitted until 2026-07-10 19:31:27.709320223 +0000 UTC m=+1974.007671570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls") pod "error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" (UID: "b86ae9c0-0709-410b-82ee-42846b74d896") : secret "error-404-isvc-1e1e7-predictor-serving-cert" not found
Jul 10 19:31:27.210245 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.210219    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.218362 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.218330    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h57h\" (UniqueName: \"kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.512117 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.512085    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.514617 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.514593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") pod \"success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") " pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.714361 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.714328    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.716812 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.716785    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") pod \"error-404-isvc-1e1e7-predictor-98b49b564-fwt8l\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") " pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.750675 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.750642    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:27.874659 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.874631    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:27.889258 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.889226    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:31:27.893046 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:31:27.893016    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4134c2_98cf_4853_b50d_30ac551f5683.slice/crio-efa1460422c6e88c2b1cf7d97131d0cd4f25d34bd2615bb4454658b9d55198d6 WatchSource:0}: Error finding container efa1460422c6e88c2b1cf7d97131d0cd4f25d34bd2615bb4454658b9d55198d6: Status 404 returned error can't find the container with id efa1460422c6e88c2b1cf7d97131d0cd4f25d34bd2615bb4454658b9d55198d6
Jul 10 19:31:27.972336 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:27.972309    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerStarted","Data":"efa1460422c6e88c2b1cf7d97131d0cd4f25d34bd2615bb4454658b9d55198d6"}
Jul 10 19:31:28.015259 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.015227    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:31:28.021709 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:31:28.021682    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86ae9c0_0709_410b_82ee_42846b74d896.slice/crio-976ab9abbab6f3bfd676b49cd2f82c08bf01a8e3911c0b9848098b55c85c5d3b WatchSource:0}: Error finding container 976ab9abbab6f3bfd676b49cd2f82c08bf01a8e3911c0b9848098b55c85c5d3b: Status 404 returned error can't find the container with id 976ab9abbab6f3bfd676b49cd2f82c08bf01a8e3911c0b9848098b55c85c5d3b
Jul 10 19:31:28.978648 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.978613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerStarted","Data":"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"}
Jul 10 19:31:28.979136 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.978676    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerStarted","Data":"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"}
Jul 10 19:31:28.979136 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.978769    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:28.979136 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.978807    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:28.979136 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.978837    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerStarted","Data":"976ab9abbab6f3bfd676b49cd2f82c08bf01a8e3911c0b9848098b55c85c5d3b"}
Jul 10 19:31:28.980078 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.980049    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:31:28.980608 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.980587    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerStarted","Data":"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"}
Jul 10 19:31:28.980684 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.980633    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:28.980684 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.980653    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerStarted","Data":"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"}
Jul 10 19:31:28.980805 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.980692    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:28.981746 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.981708    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:31:28.996145 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:28.996105    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podStartSLOduration=2.996091341 podStartE2EDuration="2.996091341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:28.994491705 +0000 UTC m=+1975.292843073" watchObservedRunningTime="2026-07-10 19:31:28.996091341 +0000 UTC m=+1975.294442710"
Jul 10 19:31:29.012217 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:29.012157    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podStartSLOduration=3.012139972 podStartE2EDuration="3.012139972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:29.011201899 +0000 UTC m=+1975.309553270" watchObservedRunningTime="2026-07-10 19:31:29.012139972 +0000 UTC m=+1975.310491344"
Jul 10 19:31:29.677851 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:29.677809    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 10 19:31:29.795539 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:29.795505    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:31:29.985581 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:29.985480    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:31:29.985976 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:29.985617    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:31:30.622173 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.622148    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:31:30.665262 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.665238    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:31:30.739992 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.739955    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58652\" (UniqueName: \"kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652\") pod \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") "
Jul 10 19:31:30.740195 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740014    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls\") pod \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") "
Jul 10 19:31:30.740195 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740041    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config\") pod \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\" (UID: \"aadf3a41-21b5-4f85-8bad-65cce3ff530a\") "
Jul 10 19:31:30.740195 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740061    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2jt8\" (UniqueName: \"kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8\") pod \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") "
Jul 10 19:31:30.740384 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740201    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") pod \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") "
Jul 10 19:31:30.740384 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740265    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config\") pod \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\" (UID: \"9e8dabdf-ef51-425b-a17d-cfacffb419d3\") "
Jul 10 19:31:30.740492 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740419    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aadf3a41-21b5-4f85-8bad-65cce3ff530a" (UID: "aadf3a41-21b5-4f85-8bad-65cce3ff530a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:31:30.740549 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740533    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.740693 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.740632    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e8dabdf-ef51-425b-a17d-cfacffb419d3" (UID: "9e8dabdf-ef51-425b-a17d-cfacffb419d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:31:30.742343 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.742317    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aadf3a41-21b5-4f85-8bad-65cce3ff530a" (UID: "aadf3a41-21b5-4f85-8bad-65cce3ff530a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:31:30.742435 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.742358    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652" (OuterVolumeSpecName: "kube-api-access-58652") pod "aadf3a41-21b5-4f85-8bad-65cce3ff530a" (UID: "aadf3a41-21b5-4f85-8bad-65cce3ff530a"). InnerVolumeSpecName "kube-api-access-58652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:31:30.742702 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.742684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8" (OuterVolumeSpecName: "kube-api-access-n2jt8") pod "9e8dabdf-ef51-425b-a17d-cfacffb419d3" (UID: "9e8dabdf-ef51-425b-a17d-cfacffb419d3"). InnerVolumeSpecName "kube-api-access-n2jt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:31:30.742702 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.742692    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e8dabdf-ef51-425b-a17d-cfacffb419d3" (UID: "9e8dabdf-ef51-425b-a17d-cfacffb419d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:31:30.798267 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.798226    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 10 19:31:30.841599 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.841561    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadf3a41-21b5-4f85-8bad-65cce3ff530a-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.841599 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.841592    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2jt8\" (UniqueName: \"kubernetes.io/projected/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-api-access-n2jt8\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.841599 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.841601    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e8dabdf-ef51-425b-a17d-cfacffb419d3-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.841599 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.841611    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e8dabdf-ef51-425b-a17d-cfacffb419d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.841879 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.841619    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58652\" (UniqueName: \"kubernetes.io/projected/aadf3a41-21b5-4f85-8bad-65cce3ff530a-kube-api-access-58652\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:31:30.991234 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.991139    2550 generic.go:356] "Generic (PLEG): container finished" podID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerID="17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82" exitCode=0
Jul 10 19:31:30.991234 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.991199    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"
Jul 10 19:31:30.991234 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.991225    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerDied","Data":"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"}
Jul 10 19:31:30.991838 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.991291    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f" event={"ID":"9e8dabdf-ef51-425b-a17d-cfacffb419d3","Type":"ContainerDied","Data":"e97d2fe0ab1745fd728e3d8d5a14f6dcddaa2f00f7326042645ebc33926ae46d"}
Jul 10 19:31:30.991838 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.991328    2550 scope.go:117] "RemoveContainer" containerID="22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"
Jul 10 19:31:30.992891 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.992870    2550 generic.go:356] "Generic (PLEG): container finished" podID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerID="2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f" exitCode=0
Jul 10 19:31:30.993009 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.992937    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"
Jul 10 19:31:30.993081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.992938    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerDied","Data":"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"}
Jul 10 19:31:30.993081 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:30.993060    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc" event={"ID":"aadf3a41-21b5-4f85-8bad-65cce3ff530a","Type":"ContainerDied","Data":"b475ee8021db4ff9a56e6705741460265db0a34c9467b234b33b87c640f38220"}
Jul 10 19:31:31.001535 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.001506    2550 scope.go:117] "RemoveContainer" containerID="17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"
Jul 10 19:31:31.010865 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.010844    2550 scope.go:117] "RemoveContainer" containerID="22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"
Jul 10 19:31:31.011190 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:31.011171    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b\": container with ID starting with 22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b not found: ID does not exist" containerID="22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"
Jul 10 19:31:31.011237 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.011200    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b"} err="failed to get container status \"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b\": rpc error: code = NotFound desc = could not find container \"22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b\": container with ID starting with 22d5a6fb901315a1e22620c915d3efb4b782b2a08e58c762c3e6ac253b056f9b not found: ID does not exist"
Jul 10 19:31:31.011237 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.011220    2550 scope.go:117] "RemoveContainer" containerID="17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"
Jul 10 19:31:31.011487 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:31.011469    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82\": container with ID starting with 17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82 not found: ID does not exist" containerID="17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"
Jul 10 19:31:31.011547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.011496    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82"} err="failed to get container status \"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82\": rpc error: code = NotFound desc = could not find container \"17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82\": container with ID starting with 17e4ce6723479fe1e10d6c72eebf4202cb22ccbfd49ee3cbad975535ea86dc82 not found: ID does not exist"
Jul 10 19:31:31.011547 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.011518    2550 scope.go:117] "RemoveContainer" containerID="132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"
Jul 10 19:31:31.019516 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.019491    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:31:31.020113 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.020099    2550 scope.go:117] "RemoveContainer" containerID="2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"
Jul 10 19:31:31.024946 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.024924    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b29b-predictor-7c6b8dbc64-6jf7f"]
Jul 10 19:31:31.027850 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.027826    2550 scope.go:117] "RemoveContainer" containerID="132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"
Jul 10 19:31:31.028077 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:31.028061    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b\": container with ID starting with 132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b not found: ID does not exist" containerID="132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"
Jul 10 19:31:31.028130 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.028083    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b"} err="failed to get container status \"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b\": rpc error: code = NotFound desc = could not find container \"132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b\": container with ID starting with 132e79867342298c13f91bcb3826d6b1cef103fbf10ae9fa51fab2af0a6df96b not found: ID does not exist"
Jul 10 19:31:31.028130 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.028099    2550 scope.go:117] "RemoveContainer" containerID="2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"
Jul 10 19:31:31.028343 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:31:31.028326    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f\": container with ID starting with 2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f not found: ID does not exist" containerID="2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"
Jul 10 19:31:31.028389 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.028348    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f"} err="failed to get container status \"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f\": rpc error: code = NotFound desc = could not find container \"2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f\": container with ID starting with 2b2a0ae981ba924a385339e9b821a04c9c0cadcaae13edacdc25f037b051e44f not found: ID does not exist"
Jul 10 19:31:31.035399 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.035376    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:31:31.039259 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:31.039239    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b29b-predictor-68c4fbfdb9-txgpc"]
Jul 10 19:31:32.274798 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:32.274765    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" path="/var/lib/kubelet/pods/9e8dabdf-ef51-425b-a17d-cfacffb419d3/volumes"
Jul 10 19:31:32.275358 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:32.275339    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" path="/var/lib/kubelet/pods/aadf3a41-21b5-4f85-8bad-65cce3ff530a/volumes"
Jul 10 19:31:34.989999 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:34.989969    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:31:34.990542 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:34.990520    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:31:34.990597 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:34.990564    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:31:34.990926 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:34.990900    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:31:40.798679 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:40.798646    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:31:44.995114 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:44.995066    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:31:44.995494 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:44.995162    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:31:54.990717 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:54.990678    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:31:54.991147 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:31:54.991028    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:32:04.990693 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:32:04.990647    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 10 19:32:04.991294 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:32:04.990967    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 10 19:32:14.991919 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:32:14.991884    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:32:14.992342 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:32:14.991975    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:40:41.584063 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.584029    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:40:41.584575 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.584266    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container" containerID="cri-o://23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02" gracePeriod=30
Jul 10 19:40:41.584575 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.584332    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kube-rbac-proxy" containerID="cri-o://14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298" gracePeriod=30
Jul 10 19:40:41.639551 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.639521    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:40:41.639791 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.639759    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container" containerID="cri-o://3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc" gracePeriod=30
Jul 10 19:40:41.639928 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:41.639773    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kube-rbac-proxy" containerID="cri-o://5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078" gracePeriod=30
Jul 10 19:40:42.069699 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:42.069664    2550 generic.go:356] "Generic (PLEG): container finished" podID="b86ae9c0-0709-410b-82ee-42846b74d896" containerID="5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078" exitCode=2
Jul 10 19:40:42.069906 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:42.069760    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerDied","Data":"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"}
Jul 10 19:40:42.071463 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:42.071442    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerID="14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298" exitCode=2
Jul 10 19:40:42.071582 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:42.071478    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerDied","Data":"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"}
Jul 10 19:40:44.751259 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.751198    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:40:44.792011 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.791985    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:40:44.829308 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829276    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") pod \"4c4134c2-98cf-4853-b50d-30ac551f5683\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") "
Jul 10 19:40:44.829484 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829317    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-669xm\" (UniqueName: \"kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm\") pod \"4c4134c2-98cf-4853-b50d-30ac551f5683\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") "
Jul 10 19:40:44.829484 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829334    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h57h\" (UniqueName: \"kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h\") pod \"b86ae9c0-0709-410b-82ee-42846b74d896\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") "
Jul 10 19:40:44.829484 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829359    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") pod \"b86ae9c0-0709-410b-82ee-42846b74d896\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") "
Jul 10 19:40:44.829484 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829383    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config\") pod \"b86ae9c0-0709-410b-82ee-42846b74d896\" (UID: \"b86ae9c0-0709-410b-82ee-42846b74d896\") "
Jul 10 19:40:44.830293 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829504    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config\") pod \"4c4134c2-98cf-4853-b50d-30ac551f5683\" (UID: \"4c4134c2-98cf-4853-b50d-30ac551f5683\") "
Jul 10 19:40:44.830293 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829777    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b86ae9c0-0709-410b-82ee-42846b74d896" (UID: "b86ae9c0-0709-410b-82ee-42846b74d896"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:40:44.830293 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.829903    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c4134c2-98cf-4853-b50d-30ac551f5683" (UID: "4c4134c2-98cf-4853-b50d-30ac551f5683"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:40:44.831571 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.831542    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm" (OuterVolumeSpecName: "kube-api-access-669xm") pod "4c4134c2-98cf-4853-b50d-30ac551f5683" (UID: "4c4134c2-98cf-4853-b50d-30ac551f5683"). InnerVolumeSpecName "kube-api-access-669xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:40:44.831571 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.831561    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c4134c2-98cf-4853-b50d-30ac551f5683" (UID: "4c4134c2-98cf-4853-b50d-30ac551f5683"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:40:44.832049 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.832028    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h" (OuterVolumeSpecName: "kube-api-access-2h57h") pod "b86ae9c0-0709-410b-82ee-42846b74d896" (UID: "b86ae9c0-0709-410b-82ee-42846b74d896"). InnerVolumeSpecName "kube-api-access-2h57h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:40:44.832049 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.832030    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b86ae9c0-0709-410b-82ee-42846b74d896" (UID: "b86ae9c0-0709-410b-82ee-42846b74d896"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:40:44.930611 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930556    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c4134c2-98cf-4853-b50d-30ac551f5683-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:44.930611 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930603    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-669xm\" (UniqueName: \"kubernetes.io/projected/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-api-access-669xm\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:44.930611 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930618    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h57h\" (UniqueName: \"kubernetes.io/projected/b86ae9c0-0709-410b-82ee-42846b74d896-kube-api-access-2h57h\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:44.930883 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930631    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b86ae9c0-0709-410b-82ee-42846b74d896-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:44.930883 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930644    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b86ae9c0-0709-410b-82ee-42846b74d896-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:44.930883 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:44.930656    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c4134c2-98cf-4853-b50d-30ac551f5683-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:40:45.083648 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.083564    2550 generic.go:356] "Generic (PLEG): container finished" podID="b86ae9c0-0709-410b-82ee-42846b74d896" containerID="3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc" exitCode=0
Jul 10 19:40:45.083648 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.083612    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"
Jul 10 19:40:45.083892 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.083661    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerDied","Data":"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"}
Jul 10 19:40:45.083892 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.083727    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l" event={"ID":"b86ae9c0-0709-410b-82ee-42846b74d896","Type":"ContainerDied","Data":"976ab9abbab6f3bfd676b49cd2f82c08bf01a8e3911c0b9848098b55c85c5d3b"}
Jul 10 19:40:45.083892 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.083765    2550 scope.go:117] "RemoveContainer" containerID="5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"
Jul 10 19:40:45.085341 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.085317    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerID="23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02" exitCode=0
Jul 10 19:40:45.085468 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.085372    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerDied","Data":"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"}
Jul 10 19:40:45.085468 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.085415    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49" event={"ID":"4c4134c2-98cf-4853-b50d-30ac551f5683","Type":"ContainerDied","Data":"efa1460422c6e88c2b1cf7d97131d0cd4f25d34bd2615bb4454658b9d55198d6"}
Jul 10 19:40:45.085585 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.085547    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"
Jul 10 19:40:45.095780 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.095750    2550 scope.go:117] "RemoveContainer" containerID="3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"
Jul 10 19:40:45.102942 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.102926    2550 scope.go:117] "RemoveContainer" containerID="5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"
Jul 10 19:40:45.103195 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:40:45.103177    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078\": container with ID starting with 5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078 not found: ID does not exist" containerID="5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"
Jul 10 19:40:45.103240 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.103203    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078"} err="failed to get container status \"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078\": rpc error: code = NotFound desc = could not find container \"5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078\": container with ID starting with 5d9376276817814f508277d3c07e1a3241d7575d6bce321d1284be20c41f8078 not found: ID does not exist"
Jul 10 19:40:45.103240 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.103219    2550 scope.go:117] "RemoveContainer" containerID="3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"
Jul 10 19:40:45.103469 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:40:45.103450    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc\": container with ID starting with 3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc not found: ID does not exist" containerID="3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"
Jul 10 19:40:45.103523 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.103474    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc"} err="failed to get container status \"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc\": rpc error: code = NotFound desc = could not find container \"3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc\": container with ID starting with 3e48c1a948c30ed8d18e534126e60b0584672e8b948759212a7fa4a01b26e9dc not found: ID does not exist"
Jul 10 19:40:45.103523 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.103487    2550 scope.go:117] "RemoveContainer" containerID="14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"
Jul 10 19:40:45.106560 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.106540    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:40:45.110847 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.110839    2550 scope.go:117] "RemoveContainer" containerID="23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"
Jul 10 19:40:45.110920 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.110895    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e1e7-predictor-98b49b564-fwt8l"]
Jul 10 19:40:45.117488 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.117472    2550 scope.go:117] "RemoveContainer" containerID="14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"
Jul 10 19:40:45.117722 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:40:45.117704    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298\": container with ID starting with 14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298 not found: ID does not exist" containerID="14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"
Jul 10 19:40:45.117788 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.117729    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298"} err="failed to get container status \"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298\": rpc error: code = NotFound desc = could not find container \"14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298\": container with ID starting with 14a36a5da705d395808b6541d9d6ef916a6372676695d1049e2fb23f38252298 not found: ID does not exist"
Jul 10 19:40:45.117788 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.117771    2550 scope.go:117] "RemoveContainer" containerID="23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"
Jul 10 19:40:45.117983 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:40:45.117965    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02\": container with ID starting with 23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02 not found: ID does not exist" containerID="23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"
Jul 10 19:40:45.118042 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.117991    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02"} err="failed to get container status \"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02\": rpc error: code = NotFound desc = could not find container \"23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02\": container with ID starting with 23d6c9dde9957538e49a9616fc7c7298e059014ccf5fe4a4d0e4a4d65dc6be02 not found: ID does not exist"
Jul 10 19:40:45.121048 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.121030    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:40:45.125029 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:45.125009    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1e1e7-predictor-b8d4c67bf-drq49"]
Jul 10 19:40:46.274575 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:46.274544    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" path="/var/lib/kubelet/pods/4c4134c2-98cf-4853-b50d-30ac551f5683/volumes"
Jul 10 19:40:46.274988 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:40:46.274974    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" path="/var/lib/kubelet/pods/b86ae9c0-0709-410b-82ee-42846b74d896/volumes"
Jul 10 19:48:01.943879 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.943797    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:48:01.946288 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.944034    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container" containerID="cri-o://ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6" gracePeriod=30
Jul 10 19:48:01.946288 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.944066    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kube-rbac-proxy" containerID="cri-o://2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b" gracePeriod=30
Jul 10 19:48:01.989330 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.989296    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:48:01.989582 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.989539    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container" containerID="cri-o://8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50" gracePeriod=30
Jul 10 19:48:01.989668 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:01.989616    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kube-rbac-proxy" containerID="cri-o://eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94" gracePeriod=30
Jul 10 19:48:02.702436 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:02.702399    2550 generic.go:356] "Generic (PLEG): container finished" podID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerID="2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b" exitCode=2
Jul 10 19:48:02.702638 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:02.702470    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerDied","Data":"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"}
Jul 10 19:48:02.704345 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:02.704311    2550 generic.go:356] "Generic (PLEG): container finished" podID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerID="eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94" exitCode=2
Jul 10 19:48:02.704465 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:02.704348    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerDied","Data":"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"}
Jul 10 19:48:04.790778 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:04.790703    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 10 19:48:05.148506 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.148483    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:48:05.151102 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.151085    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:48:05.201518 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201482    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config\") pod \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") "
Jul 10 19:48:05.201707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201544    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msc4f\" (UniqueName: \"kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f\") pod \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") "
Jul 10 19:48:05.201707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201568    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config\") pod \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") "
Jul 10 19:48:05.201707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201599    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls\") pod \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\" (UID: \"5992cd9f-e127-4fe4-80d1-6257a5b4e44c\") "
Jul 10 19:48:05.201707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201631    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") pod \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") "
Jul 10 19:48:05.201707 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201658    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbw2m\" (UniqueName: \"kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m\") pod \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\" (UID: \"d43689cc-70b7-41fd-9e17-a1f9971ae6ee\") "
Jul 10 19:48:05.201969 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201895    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5992cd9f-e127-4fe4-80d1-6257a5b4e44c" (UID: "5992cd9f-e127-4fe4-80d1-6257a5b4e44c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:48:05.202021 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.201997    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d43689cc-70b7-41fd-9e17-a1f9971ae6ee" (UID: "d43689cc-70b7-41fd-9e17-a1f9971ae6ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:48:05.204006 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.203972    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d43689cc-70b7-41fd-9e17-a1f9971ae6ee" (UID: "d43689cc-70b7-41fd-9e17-a1f9971ae6ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:48:05.204124 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.204006    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f" (OuterVolumeSpecName: "kube-api-access-msc4f") pod "5992cd9f-e127-4fe4-80d1-6257a5b4e44c" (UID: "5992cd9f-e127-4fe4-80d1-6257a5b4e44c"). InnerVolumeSpecName "kube-api-access-msc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:48:05.204124 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.204020    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5992cd9f-e127-4fe4-80d1-6257a5b4e44c" (UID: "5992cd9f-e127-4fe4-80d1-6257a5b4e44c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:48:05.204124 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.204020    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m" (OuterVolumeSpecName: "kube-api-access-zbw2m") pod "d43689cc-70b7-41fd-9e17-a1f9971ae6ee" (UID: "d43689cc-70b7-41fd-9e17-a1f9971ae6ee"). InnerVolumeSpecName "kube-api-access-zbw2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302506    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302536    2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-proxy-tls\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302545    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbw2m\" (UniqueName: \"kubernetes.io/projected/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-api-access-zbw2m\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302553    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302563    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msc4f\" (UniqueName: \"kubernetes.io/projected/5992cd9f-e127-4fe4-80d1-6257a5b4e44c-kube-api-access-msc4f\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.302591 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.302571    2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d43689cc-70b7-41fd-9e17-a1f9971ae6ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-20.ec2.internal\" DevicePath \"\""
Jul 10 19:48:05.717090 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.717057    2550 generic.go:356] "Generic (PLEG): container finished" podID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerID="ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6" exitCode=0
Jul 10 19:48:05.717279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.717110    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"
Jul 10 19:48:05.717279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.717147    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerDied","Data":"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"}
Jul 10 19:48:05.717279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.717214    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr" event={"ID":"5992cd9f-e127-4fe4-80d1-6257a5b4e44c","Type":"ContainerDied","Data":"715e9ab1ff3bfb959acf086cfc1fb65322607bebed76fe713b7870be2e02ab13"}
Jul 10 19:48:05.717279 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.717258    2550 scope.go:117] "RemoveContainer" containerID="2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"
Jul 10 19:48:05.718935 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.718917    2550 generic.go:356] "Generic (PLEG): container finished" podID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerID="8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50" exitCode=0
Jul 10 19:48:05.719096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.718968    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerDied","Data":"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"}
Jul 10 19:48:05.719096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.719008    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46" event={"ID":"d43689cc-70b7-41fd-9e17-a1f9971ae6ee","Type":"ContainerDied","Data":"f84ec0a5bfde94705a297476cb07fa4d97efb2aff5cb3fd014b75ff3b5c342e3"}
Jul 10 19:48:05.719096 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.719009    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"
Jul 10 19:48:05.727000 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.726982    2550 scope.go:117] "RemoveContainer" containerID="ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"
Jul 10 19:48:05.734158 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.734143    2550 scope.go:117] "RemoveContainer" containerID="2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"
Jul 10 19:48:05.734415 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:48:05.734397    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b\": container with ID starting with 2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b not found: ID does not exist" containerID="2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"
Jul 10 19:48:05.734486 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.734425    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b"} err="failed to get container status \"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b\": rpc error: code = NotFound desc = could not find container \"2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b\": container with ID starting with 2972a2eaa30a9cb4fde3b59837802210afb180410f0ebfc7006c3af7612c0e3b not found: ID does not exist"
Jul 10 19:48:05.734486 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.734448    2550 scope.go:117] "RemoveContainer" containerID="ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"
Jul 10 19:48:05.734718 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:48:05.734702    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6\": container with ID starting with ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6 not found: ID does not exist" containerID="ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"
Jul 10 19:48:05.734784 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.734724    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6"} err="failed to get container status \"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6\": rpc error: code = NotFound desc = could not find container \"ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6\": container with ID starting with ce4ca0edfce351a3b9dafa6cc6bede11c062b84131b0b71e2c791fc33fca7ac6 not found: ID does not exist"
Jul 10 19:48:05.734784 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.734762    2550 scope.go:117] "RemoveContainer" containerID="eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"
Jul 10 19:48:05.738515 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.738472    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:48:05.740492 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.740473    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c787-predictor-5b4bb49b7-8qlrr"]
Jul 10 19:48:05.742835 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.742813    2550 scope.go:117] "RemoveContainer" containerID="8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"
Jul 10 19:48:05.749621 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.749605    2550 scope.go:117] "RemoveContainer" containerID="eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"
Jul 10 19:48:05.749937 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:48:05.749911    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94\": container with ID starting with eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94 not found: ID does not exist" containerID="eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"
Jul 10 19:48:05.750026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.749946    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94"} err="failed to get container status \"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94\": rpc error: code = NotFound desc = could not find container \"eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94\": container with ID starting with eeadcde9d0d791de3c80126860a4b0f64b0eaa8bcada207eb6849fadd1c10e94 not found: ID does not exist"
Jul 10 19:48:05.750026 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.749969    2550 scope.go:117] "RemoveContainer" containerID="8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"
Jul 10 19:48:05.750382 ip-10-0-139-20 kubenswrapper[2550]: E0710 19:48:05.750353    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50\": container with ID starting with 8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50 not found: ID does not exist" containerID="8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"
Jul 10 19:48:05.750491 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.750399    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50"} err="failed to get container status \"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50\": rpc error: code = NotFound desc = could not find container \"8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50\": container with ID starting with 8ab9f205c60454e339fadded6c27e8b6c9e17642944b6ef5291ae7218d91fb50 not found: ID does not exist"
Jul 10 19:48:05.751714 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.751697    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:48:05.755088 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:05.755071    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c787-predictor-794854c758-r7j46"]
Jul 10 19:48:06.274927 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:06.274896    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" path="/var/lib/kubelet/pods/5992cd9f-e127-4fe4-80d1-6257a5b4e44c/volumes"
Jul 10 19:48:06.275296 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:06.275289    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" path="/var/lib/kubelet/pods/d43689cc-70b7-41fd-9e17-a1f9971ae6ee/volumes"
Jul 10 19:48:30.188706 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.188667    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fqzzt/must-gather-nl62q"]
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.188973    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.188985    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.188994    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189005    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189018    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189025    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189030    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189035    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189042    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189047    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189055    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189060    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189072    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189078    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189086    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189094    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189102    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189106    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189114    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189118    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189126    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189132    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189137    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189135 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189142    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189195    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189204    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189209    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189214    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189220    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189226    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5992cd9f-e127-4fe4-80d1-6257a5b4e44c" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189231    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189236    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8dabdf-ef51-425b-a17d-cfacffb419d3" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189244    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadf3a41-21b5-4f85-8bad-65cce3ff530a" containerName="kube-rbac-proxy"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189253    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43689cc-70b7-41fd-9e17-a1f9971ae6ee" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189261    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4134c2-98cf-4853-b50d-30ac551f5683" containerName="kserve-container"
Jul 10 19:48:30.189856 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.189266    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86ae9c0-0709-410b-82ee-42846b74d896" containerName="kube-rbac-proxy"
Jul 10 19:48:30.191336 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.191318    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.193761 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.193721    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqzzt\"/\"openshift-service-ca.crt\""
Jul 10 19:48:30.193904 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.193835    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqzzt\"/\"kube-root-ca.crt\""
Jul 10 19:48:30.193904 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.193877    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fqzzt\"/\"default-dockercfg-vr2dg\""
Jul 10 19:48:30.202861 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.202821    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqzzt/must-gather-nl62q"]
Jul 10 19:48:30.271191 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.271149    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-must-gather-output\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.271362 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.271249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vsq\" (UniqueName: \"kubernetes.io/projected/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-kube-api-access-n4vsq\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.372057 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.372016    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vsq\" (UniqueName: \"kubernetes.io/projected/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-kube-api-access-n4vsq\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.372228 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.372073    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-must-gather-output\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.372377 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.372361    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-must-gather-output\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.380461 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.380434    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vsq\" (UniqueName: \"kubernetes.io/projected/fc6f9071-7755-4ccd-89f6-9f4d9723a85c-kube-api-access-n4vsq\") pod \"must-gather-nl62q\" (UID: \"fc6f9071-7755-4ccd-89f6-9f4d9723a85c\") " pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.500787 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.500653    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqzzt/must-gather-nl62q"
Jul 10 19:48:30.639926 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.639894    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqzzt/must-gather-nl62q"]
Jul 10 19:48:30.642834 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:48:30.642798    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc6f9071_7755_4ccd_89f6_9f4d9723a85c.slice/crio-abfb0d4698223a106dfefaeb3992016617bc01940a993b57cf84009c4e6bc6ab WatchSource:0}: Error finding container abfb0d4698223a106dfefaeb3992016617bc01940a993b57cf84009c4e6bc6ab: Status 404 returned error can't find the container with id abfb0d4698223a106dfefaeb3992016617bc01940a993b57cf84009c4e6bc6ab
Jul 10 19:48:30.644824 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.644806    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:48:30.815701 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:30.815613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqzzt/must-gather-nl62q" event={"ID":"fc6f9071-7755-4ccd-89f6-9f4d9723a85c","Type":"ContainerStarted","Data":"abfb0d4698223a106dfefaeb3992016617bc01940a993b57cf84009c4e6bc6ab"}
Jul 10 19:48:31.823265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:31.822418    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqzzt/must-gather-nl62q" event={"ID":"fc6f9071-7755-4ccd-89f6-9f4d9723a85c","Type":"ContainerStarted","Data":"c286ebddb9767fc99c4be5cc4b9599b350d2b1b338bbebc0b3a59ed096aed61c"}
Jul 10 19:48:31.823265 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:31.822506    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqzzt/must-gather-nl62q" event={"ID":"fc6f9071-7755-4ccd-89f6-9f4d9723a85c","Type":"ContainerStarted","Data":"5cedb0024a1fa9bacb349141aa731495b4a61a5897d3c970974cfac1208263c1"}
Jul 10 19:48:31.840434 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:31.840390    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fqzzt/must-gather-nl62q" podStartSLOduration=1.1914838429999999 podStartE2EDuration="1.840372908s" totalImagesPullingTime="648.889065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:48:31.83988346 +0000 UTC m=+2998.138234829" watchObservedRunningTime="2026-07-10 19:48:31.840372908 +0000 UTC m=+2998.138724276"
Jul 10 19:48:32.668455 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:32.668427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w9gm9_537fede7-1201-4688-9690-276e542b8f13/global-pull-secret-syncer/0.log"
Jul 10 19:48:32.930505 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:32.930438    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tz7r4_d42dd62c-6e03-4e8d-bc39-3021e33abd05/konnectivity-agent/0.log"
Jul 10 19:48:33.005124 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:33.005093    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-20.ec2.internal_1d70378dbc438c3228b7eaf66f8d5709/haproxy/0.log"
Jul 10 19:48:37.296620 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:37.296588    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-2httj_2dce1169-5d8a-46c0-ac53-31fe92c82eae/monitoring-plugin/0.log"
Jul 10 19:48:37.332230 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:37.332195    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lfbz_01d4242f-5edc-40ed-982f-3b8ce5e92e0a/node-exporter/0.log"
Jul 10 19:48:37.354809 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:37.354773    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lfbz_01d4242f-5edc-40ed-982f-3b8ce5e92e0a/kube-rbac-proxy/0.log"
Jul 10 19:48:37.381332 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:37.381301    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lfbz_01d4242f-5edc-40ed-982f-3b8ce5e92e0a/init-textfile/0.log"
Jul 10 19:48:38.177370 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.177339    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/thanos-query/0.log"
Jul 10 19:48:38.205317 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.205289    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/kube-rbac-proxy-web/0.log"
Jul 10 19:48:38.232290 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.232222    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/kube-rbac-proxy/0.log"
Jul 10 19:48:38.267687 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.267657    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/prom-label-proxy/0.log"
Jul 10 19:48:38.298261 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.298219    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/kube-rbac-proxy-rules/0.log"
Jul 10 19:48:38.327203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:38.327167    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f4cb5799-vd4h6_5d184c2b-5f86-40f4-8b51-0c94bccf42a5/kube-rbac-proxy-metrics/0.log"
Jul 10 19:48:39.628813 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:39.628787    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-2wg4d_fe7f67ed-0c28-4b62-8b30-2add5ab2412e/networking-console-plugin/0.log"
Jul 10 19:48:40.860275 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.858882    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"]
Jul 10 19:48:40.864755 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.864714    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:40.872751 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.872702    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"]
Jul 10 19:48:40.965010 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.964971    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-lib-modules\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:40.965203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.965061    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-sys\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:40.965203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.965091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgmp\" (UniqueName: \"kubernetes.io/projected/7a8f0fab-993d-44f8-823f-b84b202ae1b2-kube-api-access-6rgmp\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:40.965203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.965131    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-podres\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:40.965203 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:40.965163    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-proc\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.066235 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-lib-modules\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.066490 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066469    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-sys\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.066606 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066591    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgmp\" (UniqueName: \"kubernetes.io/projected/7a8f0fab-993d-44f8-823f-b84b202ae1b2-kube-api-access-6rgmp\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066704    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-podres\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066759    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-proc\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066867    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-proc\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066496    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-lib-modules\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.066553    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-sys\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.067320 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.067291    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a8f0fab-993d-44f8-823f-b84b202ae1b2-podres\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.076462 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.076431    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgmp\" (UniqueName: \"kubernetes.io/projected/7a8f0fab-993d-44f8-823f-b84b202ae1b2-kube-api-access-6rgmp\") pod \"perf-node-gather-daemonset-xqbc9\" (UID: \"7a8f0fab-993d-44f8-823f-b84b202ae1b2\") " pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.177286 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.177248    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:41.346109 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.346073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"]
Jul 10 19:48:41.353638 ip-10-0-139-20 kubenswrapper[2550]: W0710 19:48:41.353592    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a8f0fab_993d_44f8_823f_b84b202ae1b2.slice/crio-f0d84d926d1462dfac577b80bb399af9c86b4d2576ee88ca25a67c3c7e3a1088 WatchSource:0}: Error finding container f0d84d926d1462dfac577b80bb399af9c86b4d2576ee88ca25a67c3c7e3a1088: Status 404 returned error can't find the container with id f0d84d926d1462dfac577b80bb399af9c86b4d2576ee88ca25a67c3c7e3a1088
Jul 10 19:48:41.874406 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.874322    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9" event={"ID":"7a8f0fab-993d-44f8-823f-b84b202ae1b2","Type":"ContainerStarted","Data":"233739570fc0fd9dcb7f04ff97e8ff70f835e831e50bc626916aa86c12d55170"}
Jul 10 19:48:41.875018 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.874982    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9" event={"ID":"7a8f0fab-993d-44f8-823f-b84b202ae1b2","Type":"ContainerStarted","Data":"f0d84d926d1462dfac577b80bb399af9c86b4d2576ee88ca25a67c3c7e3a1088"}
Jul 10 19:48:41.875165 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:41.875043    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:42.007813 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:42.007767    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gm5jl_ae930abd-fa30-48d4-a391-403d4d3c3073/dns/0.log"
Jul 10 19:48:42.034518 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:42.034476    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gm5jl_ae930abd-fa30-48d4-a391-403d4d3c3073/kube-rbac-proxy/0.log"
Jul 10 19:48:42.163252 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:42.163222    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jmnrl_a6b2a3f9-d5b8-4242-a5e3-b1b5fd6dc422/dns-node-resolver/0.log"
Jul 10 19:48:42.726352 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:42.726316    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsw4k_9b103d35-f097-45f6-906b-11016846795f/node-ca/0.log"
Jul 10 19:48:43.954649 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:43.954615    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ww9lx_50b1ee8c-3513-412b-bce4-68e0dcfe1b79/serve-healthcheck-canary/0.log"
Jul 10 19:48:44.468837 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:44.468810    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qslt6_4867a0c0-f343-4ff2-bdf9-e4dbc9be4761/kube-rbac-proxy/0.log"
Jul 10 19:48:44.490334 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:44.490312    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qslt6_4867a0c0-f343-4ff2-bdf9-e4dbc9be4761/exporter/0.log"
Jul 10 19:48:44.511449 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:44.511427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qslt6_4867a0c0-f343-4ff2-bdf9-e4dbc9be4761/extractor/0.log"
Jul 10 19:48:46.607528 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:46.607502    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-rhf8n_6306cc0c-bfb1-4336-81b8-01d6e34773c7/server/0.log"
Jul 10 19:48:46.723648 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:46.723617    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-wr4nb_96f38207-8fdf-441b-95a4-ac367975d26f/manager/0.log"
Jul 10 19:48:46.793502 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:46.793473    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-b5vkl_3395d696-33cd-431e-8006-b81f52bb1377/seaweedfs/0.log"
Jul 10 19:48:47.887871 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:47.887840    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9"
Jul 10 19:48:47.916134 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:47.916078    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fqzzt/perf-node-gather-daemonset-xqbc9" podStartSLOduration=7.916058306 podStartE2EDuration="7.916058306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:48:41.894395373 +0000 UTC m=+3008.192746743" watchObservedRunningTime="2026-07-10 19:48:47.916058306 +0000 UTC m=+3014.214409676"
Jul 10 19:48:52.652221 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:52.652183    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4qbhw_39adceed-d61c-41d6-abf8-2c341e30d3f9/kube-multus/0.log"
Jul 10 19:48:53.084002 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.083972    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:48:53.133875 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.133842    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/egress-router-binary-copy/0.log"
Jul 10 19:48:53.161307 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.161285    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/cni-plugins/0.log"
Jul 10 19:48:53.194868 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.194822    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/bond-cni-plugin/0.log"
Jul 10 19:48:53.234725 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.234701    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/routeoverride-cni/0.log"
Jul 10 19:48:53.290786 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.290756    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/whereabouts-cni-bincopy/0.log"
Jul 10 19:48:53.351335 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.351264    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pzkb7_5691beb4-1cf8-4df5-b6a4-f11bb372c145/whereabouts-cni/0.log"
Jul 10 19:48:53.629070 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.629002    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5mrgz_bad76711-2921-4515-b2f4-26554d0783aa/network-metrics-daemon/0.log"
Jul 10 19:48:53.666819 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:53.666794    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5mrgz_bad76711-2921-4515-b2f4-26554d0783aa/kube-rbac-proxy/0.log"
Jul 10 19:48:54.998354 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:54.998324    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/ovn-controller/0.log"
Jul 10 19:48:55.021563 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.021543    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/ovn-acl-logging/0.log"
Jul 10 19:48:55.048992 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.048972    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/kube-rbac-proxy-node/0.log"
Jul 10 19:48:55.077004 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.076982    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:48:55.105396 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.105365    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/northd/0.log"
Jul 10 19:48:55.139045 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.139023    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/nbdb/0.log"
Jul 10 19:48:55.165670 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.165645    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/sbdb/0.log"
Jul 10 19:48:55.225539 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:55.225517    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9x886_56678559-0a6b-42b5-b192-5d8bcd4a36ed/ovnkube-controller/0.log"
Jul 10 19:48:56.681847 ip-10-0-139-20 kubenswrapper[2550]: I0710 19:48:56.681821    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sfhhk_93732cb3-2584-40f5-babc-d6e086652f69/network-check-target-container/0.log"