Jul 10 18:56:33.363996 ip-10-0-134-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 18:56:33.364012 ip-10-0-134-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 18:56:33.364021 ip-10-0-134-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 18:56:33.364482 ip-10-0-134-26 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 18:56:43.442109 ip-10-0-134-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 18:56:43.442164 ip-10-0-134-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5b6ff41d91c34120a004242661e566c2 --
Jul 10 18:58:35.742152 ip-10-0-134-26 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.183486    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193425    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193444    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193449    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193452    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193456    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193460    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193463    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.216083 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193466    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193468    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193472    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193475    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193478    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193481    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193483    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193486    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193489    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193492    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193494    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193497    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193500    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193503    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193505    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193508    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193511    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193513    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193516    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193521    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.216883 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193524    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193527    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193529    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193532    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193534    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193537    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193540    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193543    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193546    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193549    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193553    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193558    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193561    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193564    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193567    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193570    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193573    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193576    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193579    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193583    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.217425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193587    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193590    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193593    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193596    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193599    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193602    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193605    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193608    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193611    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193613    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193616    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193619    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193621    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193624    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193626    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193629    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193631    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193634    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193636    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193639    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.217949 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193642    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193646    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193649    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193652    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193655    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193657    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193660    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193664    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193667    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193670    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193672    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193675    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193677    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193680    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193682    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193685    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193688    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193691    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193693    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.218633 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193696    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193698    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193701    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193703    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193706    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193708    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193711    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193714    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193717    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193719    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193722    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193724    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.193727    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194486    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194493    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194497    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194500    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194504    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194506    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194509    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.219194 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194511    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194514    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194516    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194519    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194521    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194524    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194527    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194530    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194532    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194535    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194538    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194540    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194543    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194545    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194549    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194553    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194556    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194559    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194561    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194564    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.219744 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194566    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194570    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194573    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194576    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194578    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194581    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194583    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194586    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194589    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194591    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194594    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194596    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194599    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194601    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194604    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194606    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194609    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194611    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194614    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194616    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.220293 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194619    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194621    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194624    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194627    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194629    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194632    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194635    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194638    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194640    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194644    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194646    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194649    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194651    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194654    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194657    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194659    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194662    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194664    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194667    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194669    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.220895 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194672    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194675    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194677    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194681    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194685    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194688    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194690    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194693    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194695    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194698    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194700    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194703    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194705    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194708    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194711    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194714    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194720    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194723    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194726    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.221407 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194729    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194732    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194734    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194737    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194739    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194742    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194744    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194747    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194750    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194753    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194755    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194758    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.194760    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195173    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195178    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195181    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195184    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195187    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195190    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.221938 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195194    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195198    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195201    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195204    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195207    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195211    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195213    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195216    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195219    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195222    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195225    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195228    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195231    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195234    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195237    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195240    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195242    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195245    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195247    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195250    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.222425 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195252    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195255    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195258    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195260    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195263    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195265    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195268    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195270    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195273    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195275    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195278    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195281    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195284    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195287    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195289    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195291    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195294    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195297    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195299    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195302    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.222983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195305    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195307    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195310    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195315    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195318    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195321    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195323    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195326    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195329    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195331    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195334    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195336    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195340    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195342    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195345    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195347    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195350    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195352    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195354    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.223479 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195357    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195359    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195362    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195364    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195368    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195371    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195374    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195377    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195380    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195383    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195389    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195391    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195394    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195397    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195400    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195402    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195405    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195408    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195411    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195413    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.224065 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195416    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195418    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195421    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195423    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195426    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195428    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195431    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195433    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195436    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195438    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195441    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195443    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195446    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.195448    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195522    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195529    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195535    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195540    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195544    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195547    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195551    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:58:36.224687 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195556    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195561    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195563    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195567    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195570    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195573    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195576    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195579    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195582    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195585    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195588    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195594    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195596    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195600    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195603    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195606    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195609    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195612    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195615    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195618    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195621    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195624    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195627    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195630    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:58:36.225287 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195632    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195637    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195639    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195642    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195645    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195648    2570 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195650    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195654    2570 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195657    2570 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195661    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195664    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195667    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195670    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195673    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195676    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195679    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195682    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195684    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195688    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195692    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195695    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195698    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195701    2570 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195704    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195707    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:58:36.225977 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195710    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195713    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195716    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195719    2570 flags.go:64] FLAG: --help="false"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195722    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-134-26.ec2.internal"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195725    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195727    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195730    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195733    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195736    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195739    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195742    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195745    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195747    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195750    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195753    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195757    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195759    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195777    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195780    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195783    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195786    2570 flags.go:64] FLAG: --lock-file=""
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195789    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195792    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:58:36.226758 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195795    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195800    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195805    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195808    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195810    2570 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195813    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195817    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195820    2570 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195822    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195827    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195829    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195834    2570 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195836    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195840    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195842    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195845    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195848    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195851    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195853    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195861    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195864    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195867    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195870    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:58:36.453950 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195873    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:58:36.256999 ip-10-0-134-26 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195878    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195882    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195885    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195888    2570 flags.go:64] FLAG: --port="10250"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195891    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195894    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0350aa8ce1bf65d04"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195897    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195900    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195902    2570 flags.go:64] FLAG: --register-node="true"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195905    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195910    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195913    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195916    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195918    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195922    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195925    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195928    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195931    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195934    2570 flags.go:64] FLAG: --runonce="false"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195936    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195939    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195942    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195945    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195947    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195950    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:58:36.479590 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195953    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195956    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195959    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195962    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195964    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195967    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195970    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195973    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195977    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195982    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195985    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195987    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195992    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195995    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.195997    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196000    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196003    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196008    2570 flags.go:64] FLAG: --v="2"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196013    2570 flags.go:64] FLAG: --version="false"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196017    2570 flags.go:64] FLAG: --vmodule=""
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196021    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196024    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196134    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196138    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.484366 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196141    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196144    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196146    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196149    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196151    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196154    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196156    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196158    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196161    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196163    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196166    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196168    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196170    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196173    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196176    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196178    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196182    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196185    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196187    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.485320 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196190    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196194    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196197    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196200    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196203    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196206    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196210    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196213    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196215    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196219    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196222    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196224    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196226    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196229    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196231    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196234    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196236    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196239    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196242    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196245    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.486014 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196247    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196250    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196252    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196255    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196257    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196260    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196262    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196265    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196267    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196271    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196274    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196276    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196279    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196281    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196284    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196286    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196289    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196292    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196296    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196298    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.486900 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196301    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196303    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196306    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196309    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196312    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196316    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196319    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196322    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196325    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196327    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196330    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196332    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196335    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196338    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196340    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196342    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196345    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196348    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196350    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196353    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.487679 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196355    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196359    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196361    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196364    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196366    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196369    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196372    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196374    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196377    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196379    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196383    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196385    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196388    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196391    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196394    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196397    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196400    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.780047 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.196403    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.196444    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.203575    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.203594    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.203611    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.203616    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203683    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203689    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203692    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203695    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203698    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203701    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203704    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203707    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203710    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.781180 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203713    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203716    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203719    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203721    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203724    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203727    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203730    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203732    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203735    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203737    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203739    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203742    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203745    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203747    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203750    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203754    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203757    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203759    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203780    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203783    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.781765 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203787    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203789    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203792    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203795    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203797    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203800    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203802    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203805    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203807    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203810    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203812    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203815    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203817    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203820    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203823    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203825    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203828    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203830    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203833    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203836    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.782436 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203838    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203840    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203843    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203846    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203851    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203853    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203856    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203858    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203861    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203863    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203869    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203873    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203877    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203881    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203884    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203887    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203889    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203892    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203894    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.783113 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203897    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203899    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203902    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203905    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203907    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203909    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203912    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203915    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203917    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203920    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203922    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203925    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203928    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203930    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203933    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203935    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203938    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203945    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203948    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203950    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.783757 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203953    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203955    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203959    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203964    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203967    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203969    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203972    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203977    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203980    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203983    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.203987    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.203992    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204113    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204117    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204120    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:58:36.784446 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204123    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204126    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204129    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204131    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204134    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204136    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204139    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204141    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204144    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204147    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204149    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204151    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204154    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204157    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204161    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204164    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204166    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204168    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204171    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204173    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204176    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:58:36.784964 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204180    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204182    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204185    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204187    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204190    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204193    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204196    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204198    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204200    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204203    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204205    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204208    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204210    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204213    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204215    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204218    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204220    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204223    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204225    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204228    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:58:36.785597 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204230    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204233    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204236    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204239    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204241    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204244    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204247    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204249    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204252    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204254    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204258    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204262    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204266    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204269    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204271    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204275    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204278    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204282    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204285    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:58:36.786196 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204287    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204289    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204292    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204295    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204297    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204300    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204303    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204307    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204309    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204312    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204314    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204317    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204319    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204321    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204324    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204327    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204329    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204332    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204336    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:58:36.786782 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204338    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204341    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204343    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204346    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204348    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204350    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204354    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204357    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204359    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204362    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204364    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204367    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204370    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204372    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204375    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204378    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:58:36.787334 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:36.204380    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.204385    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.204529    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.206597    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.208046    2570 server.go:1019] "Starting client certificate rotation"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.208147    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.208186    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.210581    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.211790    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.223823    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.225945    2570 log.go:25] "Validated CRI v1 image API"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.226707    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:58:36.787854 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.229624    2570 fs.go:135] Filesystem UUIDs: map[031c573b-57da-4ecf-a628-ec8808cea3b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae32de69-43eb-4995-a0f5-1d967af61021:/dev/nvme0n1p4]
Jul 10 18:58:36.788227 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.229638    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.236589    2570 manager.go:216] Machine: {Timestamp:2026-07-10 18:58:36.235181131 +0000 UTC m=+0.369111789 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200289 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d87811edf958964a8c903ba270ed SystemUUID:ec23d878-11ed-f958-964a-8c903ba270ed BootID:5b6ff41d-91c3-4120-a004-242661e566c2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:78:b2:05:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:78:b2:05:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:c3:b1:bb:c5:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.236691    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.236917    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.239859    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.239919    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.239895    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:58:36.788346 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.240744    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.240754    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.240777    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.242364    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.243206    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246804    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246922    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246946    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246956    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246970    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246980    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.246989    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.247284    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.247297    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.249617    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251364    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251392    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251939    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251952    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251958    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251963    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251971    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.251977    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252175    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252181    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252188    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252193    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252201    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252208    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252917    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.252923    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.255624    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.255804    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.255914    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.256026    2570 server.go:1268] "Started kubelet"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.256110    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.256117    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.256272    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.257819    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.258154    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:58:36.788964 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.258797    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vq2b7"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.263626    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.263638    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.263749    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.264212    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.264265    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.264285    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.264375    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.264384    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265045    2570 factory.go:153] Registering CRI-O factory
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265063    2570 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265118    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265129    2570 factory.go:55] Registering systemd factory
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265137    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265157    2570 factory.go:103] Registering Raw factory
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265169    2570 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.265537    2570 manager.go:349] Starting recovery of all containers
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.265700    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.270360    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.270884    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.270430    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-26.ec2.internal.18c102b7e4c7e1b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-26.ec2.internal,UID:ip-10-0-134-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-26.ec2.internal,},FirstTimestamp:2026-07-10 18:58:36.256002483 +0000 UTC m=+0.389933140,LastTimestamp:2026-07-10 18:58:36.256002483 +0000 UTC m=+0.389933140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-26.ec2.internal,}"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.274324    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vq2b7"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.281037    2570 manager.go:354] Recovery completed
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.285245    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.287671    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.287702    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.287712    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288239    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288249    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288254    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288264    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288936    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:36.790225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288966    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.288980    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.290070    2570 policy_none.go:49] "None policy: Start"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.290099    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.290111    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.290062    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-26.ec2.internal.18c102b7e6ab5b25  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-26.ec2.internal,UID:ip-10-0-134-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-26.ec2.internal,},FirstTimestamp:2026-07-10 18:58:36.287687461 +0000 UTC m=+0.421618124,LastTimestamp:2026-07-10 18:58:36.287687461 +0000 UTC m=+0.421618124,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-26.ec2.internal,}"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.291937    2570 policy_none.go:47] "Start"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.342463    2570 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.342714    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.342725    2570 server.go:87] "Starting device plugin registration server"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.342983    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.342996    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.343116    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.343218    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.343229    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.343709    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.343756    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.407399    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.408626    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.408663    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.408694    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.408728    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.411170    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.443468    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.444555    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.444590    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.444604    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.444628    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.454612    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.454634    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-26.ec2.internal\": node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.485374    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.508954    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"]
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.509036    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:37.086985 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.510816    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.510843    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.510852    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513161    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513280    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513310    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513961    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513971    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513989    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.513993    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.514006    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.514035    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.516348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.516373    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.517134    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.517159    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.517172    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.549767    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-26.ec2.internal\" not found" node="ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.554558    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-26.ec2.internal\" not found" node="ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.586048    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.665880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.665935    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.665958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a274b38950da356e83a1d1b30ba640f1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"a274b38950da356e83a1d1b30ba640f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.686441    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.088677 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766204    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a274b38950da356e83a1d1b30ba640f1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"a274b38950da356e83a1d1b30ba640f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a274b38950da356e83a1d1b30ba640f1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"a274b38950da356e83a1d1b30ba640f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.766352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b969a79c02bbbfd14d439b040b8bbd48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"b969a79c02bbbfd14d439b040b8bbd48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.787457    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.855730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:36.858054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.887970    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.089425 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:36.988975    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.090297 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:37.090155    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.109913    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:37.190291    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.207729    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.207965    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.207984    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.279472    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:53:37 +0000 UTC" deadline="2028-02-09 17:00:56.712549585 +0000 UTC"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.279507    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13894h2m19.433046605s"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:37.290393    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Jul 10 18:58:37.342209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.304465    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:37.503731 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.364987    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.503731 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.377882    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:37.503731 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.379866    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Jul 10 18:58:37.503731 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.390052    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:58:37.780263 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:37.548205    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda274b38950da356e83a1d1b30ba640f1.slice/crio-2dbe9e0696eb2fdbe91dcba55d704c4f59dd5ad5b29288360c1678af897600e4 WatchSource:0}: Error finding container 2dbe9e0696eb2fdbe91dcba55d704c4f59dd5ad5b29288360c1678af897600e4: Status 404 returned error can't find the container with id 2dbe9e0696eb2fdbe91dcba55d704c4f59dd5ad5b29288360c1678af897600e4
Jul 10 18:58:37.780263 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:37.548571    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb969a79c02bbbfd14d439b040b8bbd48.slice/crio-dae7d14318bce1c0623eb2e9ab5dbfca16168d91e5bba77ad576955d06b9ed1f WatchSource:0}: Error finding container dae7d14318bce1c0623eb2e9ab5dbfca16168d91e5bba77ad576955d06b9ed1f: Status 404 returned error can't find the container with id dae7d14318bce1c0623eb2e9ab5dbfca16168d91e5bba77ad576955d06b9ed1f
Jul 10 18:58:37.780263 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.555563    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:58:37.780263 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:37.615188    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:38.249311 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.248292    2570 apiserver.go:52] "Watching apiserver"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.258191    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.258628    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal","openshift-multus/multus-hcf89","openshift-network-diagnostics/network-check-target-jg5tw","kube-system/global-pull-secret-syncer-6d6ql","kube-system/konnectivity-agent-g5rzb","openshift-dns/node-resolver-xsggr","openshift-multus/multus-additional-cni-plugins-m2xd5","openshift-multus/network-metrics-daemon-8qdk8","openshift-network-operator/iptables-alerter-9496x","openshift-ovn-kubernetes/ovnkube-node-n5mrx","kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f","openshift-cluster-node-tuning-operator/tuned-nfh97","openshift-image-registry/node-ca-kcb4c"]
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.261454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.263912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.264697    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265001    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265329    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265524    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265605    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265524    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.265884    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-849wd\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.268686    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mmr78\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.268933    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.269008    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.269214    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.269223    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.269399    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.269468    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.271187    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.273427    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.274160    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.274706    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rmxt8\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.274997    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.275827    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.366808 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.276372    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.276422    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-skzj7\""
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.276736    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.279048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.281222    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysconfig\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-conf\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285220    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-sys\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285248    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-kubernetes\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-run\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-lib-modules\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-cnibin\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.369171 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-os-release\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kfj\" (UniqueName: \"kubernetes.io/projected/b226814b-8a33-46dc-af68-bafeb83fa909-kube-api-access-25kfj\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-modprobe-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-var-lib-kubelet\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285505    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-host\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-system-cni-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-systemd\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-etc-tuned\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-tmp\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhh6\" (UniqueName: \"kubernetes.io/projected/8ab21d36-15af-4811-b62b-0acee70cee18-kube-api-access-8jhh6\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.285673    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.286700    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.286984    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.287210    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6l68k\""
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.288553    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.288623    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.370065 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.289059    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.291438    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9g2r\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.291701    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.291718    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.291870    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.291888    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.292819    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.292906    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.295451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.295697    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.295975    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mk6hz\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.296245    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.296272    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.296562    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x4lpn\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.296631    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.296962    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.297177    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.299549    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.299801    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.300187    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-64wb6\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.300442    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.312424    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f6vrh"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.322618    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f6vrh"
Jul 10 18:58:38.371395 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.366078    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-k8s-cni-cncf-io\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386136    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-daemon-config\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386376    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-socket-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386423    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-kubernetes\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-iptables-alerter-script\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-kubernetes\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386537    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-multus-certs\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.516078 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386572    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-etc-kubernetes\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386611    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-var-lib-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-log-socket\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386671    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovn-node-metrics-cert\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386699    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-os-release\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386727    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kfj\" (UniqueName: \"kubernetes.io/projected/b226814b-8a33-46dc-af68-bafeb83fa909-kube-api-access-25kfj\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-tmp\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386872    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-os-release\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-dbus\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-system-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.386978    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cni-binary-copy\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387003    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-node-log\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387019    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.518307 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387033    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-system-cni-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387071    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-systemd\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3179a044-43c4-48c9-9175-810563d50e49-hosts-file\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387133    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-host-slash\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cnibin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-slash\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-etc-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387222    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-device-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387243    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387265    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387339    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387340    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387367    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-conf\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-lib-modules\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.519180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-systemd-units\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387453    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-netns\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387481    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-run\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387559    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-ovn\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-bin\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-etc-selinux\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-sys-fs\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-cnibin\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387674    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6be89d1-7d75-49fd-b6fb-23a749d08caf-konnectivity-ca\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-os-release\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-netd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3179a044-43c4-48c9-9175-810563d50e49-tmp-dir\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-bin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.520761 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-hostroot\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvl6\" (UniqueName: \"kubernetes.io/projected/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-kube-api-access-qxvl6\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-systemd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.387987    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-env-overrides\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjq9t\" (UniqueName: \"kubernetes.io/projected/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-kube-api-access-gjq9t\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbt5\" (UniqueName: \"kubernetes.io/projected/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kube-api-access-6bbt5\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388051    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6gx\" (UniqueName: \"kubernetes.io/projected/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-kube-api-access-pf6gx\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-config\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-modprobe-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-var-lib-kubelet\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388161    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-etc-tuned\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbj4\" (UniqueName: \"kubernetes.io/projected/3179a044-43c4-48c9-9175-810563d50e49-kube-api-access-sdbj4\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-netns\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-multus\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.521675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388314    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhh6\" (UniqueName: \"kubernetes.io/projected/8ab21d36-15af-4811-b62b-0acee70cee18-kube-api-access-8jhh6\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-kubelet\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-registration-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388395    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388442    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysconfig\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-sys\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388494    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-script-lib\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3783f4ff-7409-46f9-9c40-7beea2ada6c0-host\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388538    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dw4p\" (UniqueName: \"kubernetes.io/projected/a0f255b1-7837-4251-9bca-3aa6563d4333-kube-api-access-2dw4p\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-cnibin\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-modprobe-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.388815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-var-lib-kubelet\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.523534 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.389046    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.389109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-systemd\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.389462    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b226814b-8a33-46dc-af68-bafeb83fa909-system-cni-dir\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.389864    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-d\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.389964    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysctl-conf\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-lib-modules\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390152    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-run\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-kubelet-config\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390544    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6be89d1-7d75-49fd-b6fb-23a749d08caf-agent-certs\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-kubelet\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390615    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b226814b-8a33-46dc-af68-bafeb83fa909-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390623    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-conf-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrxk\" (UniqueName: \"kubernetes.io/projected/3783f4ff-7409-46f9-9c40-7beea2ada6c0-kube-api-access-fhrxk\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.524211 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390667    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-sys\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390680    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-host\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-socket-dir-parent\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390733    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-host\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390770    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3783f4ff-7409-46f9-9c40-7beea2ada6c0-serviceca\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.390771    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab21d36-15af-4811-b62b-0acee70cee18-etc-sysconfig\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.391946    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-tmp\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.392730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab21d36-15af-4811-b62b-0acee70cee18-etc-tuned\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.401890    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.401920    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.401936    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.402036    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.901998587 +0000 UTC m=+3.035929252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.405644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhh6\" (UniqueName: \"kubernetes.io/projected/8ab21d36-15af-4811-b62b-0acee70cee18-kube-api-access-8jhh6\") pod \"tuned-nfh97\" (UID: \"8ab21d36-15af-4811-b62b-0acee70cee18\") " pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.406115    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kfj\" (UniqueName: \"kubernetes.io/projected/b226814b-8a33-46dc-af68-bafeb83fa909-kube-api-access-25kfj\") pod \"multus-additional-cni-plugins-m2xd5\" (UID: \"b226814b-8a33-46dc-af68-bafeb83fa909\") " pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.415525    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" event={"ID":"a274b38950da356e83a1d1b30ba640f1","Type":"ContainerStarted","Data":"2dbe9e0696eb2fdbe91dcba55d704c4f59dd5ad5b29288360c1678af897600e4"}
Jul 10 18:58:38.526787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.417107    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"b969a79c02bbbfd14d439b040b8bbd48","Type":"ContainerStarted","Data":"dae7d14318bce1c0623eb2e9ab5dbfca16168d91e5bba77ad576955d06b9ed1f"}
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491611    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-iptables-alerter-script\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491674    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-multus-certs\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-etc-kubernetes\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-var-lib-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491788    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-multus-certs\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-log-socket\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-etc-kubernetes\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491835    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovn-node-metrics-cert\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.491851    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-var-lib-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491904    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-log-socket\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.491857    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-dbus\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.527475 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.491982    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.991960543 +0000 UTC m=+3.125891226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-system-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cni-binary-copy\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492066    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-dbus\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492079    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-node-log\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492091    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-system-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-node-log\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3179a044-43c4-48c9-9175-810563d50e49-hosts-file\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492170    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-host-slash\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492177    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3179a044-43c4-48c9-9175-810563d50e49-hosts-file\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cnibin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-slash\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492282    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-host-slash\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492293    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-etc-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492299    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cnibin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.529549 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-device-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492320    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-iptables-alerter-script\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492343    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-etc-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-slash\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-systemd-units\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-netns\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-systemd-units\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492411    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-device-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492410    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-netns\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-openvswitch\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-ovn\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492496    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-bin\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492518    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-etc-selinux\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492538    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-ovn\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.530235 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492558    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-bin\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492590    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-sys-fs\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492619    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6be89d1-7d75-49fd-b6fb-23a749d08caf-konnectivity-ca\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492644    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-os-release\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492670    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-netd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492693    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3179a044-43c4-48c9-9175-810563d50e49-tmp-dir\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492716    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-etc-selinux\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492723    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-os-release\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492642    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-cni-binary-copy\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492670    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-sys-fs\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492757    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-cni-netd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-bin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492805    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-hostroot\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-bin\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492838    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvl6\" (UniqueName: \"kubernetes.io/projected/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-kube-api-access-qxvl6\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.532982 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492855    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-hostroot\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492862    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-systemd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-env-overrides\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492932    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-run-systemd\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.492984    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjq9t\" (UniqueName: \"kubernetes.io/projected/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-kube-api-access-gjq9t\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493013    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbt5\" (UniqueName: \"kubernetes.io/projected/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kube-api-access-6bbt5\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3179a044-43c4-48c9-9175-810563d50e49-tmp-dir\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6gx\" (UniqueName: \"kubernetes.io/projected/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-kube-api-access-pf6gx\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-config\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbj4\" (UniqueName: \"kubernetes.io/projected/3179a044-43c4-48c9-9175-810563d50e49-kube-api-access-sdbj4\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493155    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-netns\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-multus\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-kubelet\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493210    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6be89d1-7d75-49fd-b6fb-23a749d08caf-konnectivity-ca\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.533695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-registration-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493269    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-script-lib\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-cni-multus\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493294    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3783f4ff-7409-46f9-9c40-7beea2ada6c0-host\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-env-overrides\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493320    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-netns\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dw4p\" (UniqueName: \"kubernetes.io/projected/a0f255b1-7837-4251-9bca-3aa6563d4333-kube-api-access-2dw4p\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-kubelet-config\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493395    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-kubelet\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493399    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6be89d1-7d75-49fd-b6fb-23a749d08caf-agent-certs\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-kubelet\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493488    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-registration-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493498    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-conf-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrxk\" (UniqueName: \"kubernetes.io/projected/3783f4ff-7409-46f9-9c40-7beea2ada6c0-kube-api-access-fhrxk\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.536442 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.493567    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493613    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-socket-dir-parent\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.493629    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.993611499 +0000 UTC m=+3.127542160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493654    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-var-lib-kubelet\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493665    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3783f4ff-7409-46f9-9c40-7beea2ada6c0-host\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493569    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-socket-dir-parent\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493692    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-conf-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493702    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3783f4ff-7409-46f9-9c40-7beea2ada6c0-serviceca\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493726    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-k8s-cni-cncf-io\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493751    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493775    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-daemon-config\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493824    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-script-lib\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.663254 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-host-run-k8s-cni-cncf-io\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493887    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493919    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-kubelet-config\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493919    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493933    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-socket-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.493797    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovnkube-config\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.494047    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.494043    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-cni-dir\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.494114    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-socket-dir\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.494261    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3783f4ff-7409-46f9-9c40-7beea2ada6c0-serviceca\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.494361    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-multus-daemon-config\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.495258    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-ovn-node-metrics-cert\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.496372    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6be89d1-7d75-49fd-b6fb-23a749d08caf-agent-certs\") pod \"konnectivity-agent-g5rzb\" (UID: \"d6be89d1-7d75-49fd-b6fb-23a749d08caf\") " pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.665336 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.512364    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbj4\" (UniqueName: \"kubernetes.io/projected/3179a044-43c4-48c9-9175-810563d50e49-kube-api-access-sdbj4\") pod \"node-resolver-xsggr\" (UID: \"3179a044-43c4-48c9-9175-810563d50e49\") " pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.513315    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dw4p\" (UniqueName: \"kubernetes.io/projected/a0f255b1-7837-4251-9bca-3aa6563d4333-kube-api-access-2dw4p\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.513416    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvl6\" (UniqueName: \"kubernetes.io/projected/2a57bc5d-8e88-42b6-8fdf-2f262c61125a-kube-api-access-qxvl6\") pod \"multus-hcf89\" (UID: \"2a57bc5d-8e88-42b6-8fdf-2f262c61125a\") " pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.513873    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjq9t\" (UniqueName: \"kubernetes.io/projected/9a1b3146-0f9e-49f0-977d-8887ff75c7a4-kube-api-access-gjq9t\") pod \"ovnkube-node-n5mrx\" (UID: \"9a1b3146-0f9e-49f0-977d-8887ff75c7a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.517492    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbt5\" (UniqueName: \"kubernetes.io/projected/55660e51-18bc-4cc7-a0d4-16036b2d9ec1-kube-api-access-6bbt5\") pod \"aws-ebs-csi-driver-node-l284f\" (UID: \"55660e51-18bc-4cc7-a0d4-16036b2d9ec1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.517841    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6gx\" (UniqueName: \"kubernetes.io/projected/c6d1484d-d443-4c20-ae86-5eeb82fc60b4-kube-api-access-pf6gx\") pod \"iptables-alerter-9496x\" (UID: \"c6d1484d-d443-4c20-ae86-5eeb82fc60b4\") " pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.519525    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrxk\" (UniqueName: \"kubernetes.io/projected/3783f4ff-7409-46f9-9c40-7beea2ada6c0-kube-api-access-fhrxk\") pod \"node-ca-kcb4c\" (UID: \"3783f4ff-7409-46f9-9c40-7beea2ada6c0\") " pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.580769    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2xd5"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.591885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nfh97"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.610458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:38.616544    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab21d36_15af_4811_b62b_0acee70cee18.slice/crio-015a18888d120b8033eab9f1c93a96fd78ea00d36f2787cdba19821b2a133db0 WatchSource:0}: Error finding container 015a18888d120b8033eab9f1c93a96fd78ea00d36f2787cdba19821b2a133db0: Status 404 returned error can't find the container with id 015a18888d120b8033eab9f1c93a96fd78ea00d36f2787cdba19821b2a133db0
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.617722    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsggr"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:38.623321    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6be89d1_7d75_49fd_b6fb_23a749d08caf.slice/crio-417e31728d16c1bb2db545c6fda2f9ee5d20156787af23e2759b9d4b8bf6e898 WatchSource:0}: Error finding container 417e31728d16c1bb2db545c6fda2f9ee5d20156787af23e2759b9d4b8bf6e898: Status 404 returned error can't find the container with id 417e31728d16c1bb2db545c6fda2f9ee5d20156787af23e2759b9d4b8bf6e898
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.627674    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9496x"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.635963    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcb4c"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.644599    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hcf89"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.652452    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:58:38.665944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.659319    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f"
Jul 10 18:58:38.767578 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.717733    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.998766    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.998844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:38.998890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.998901    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.998981    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.998961174 +0000 UTC m=+4.132891839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999008    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999025    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999043    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999057    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999072    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.999056587 +0000 UTC m=+4.132987255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:39.037247 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:38.999093    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.999081922 +0000 UTC m=+4.133012570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:39.329958 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.329850    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:39 +0000 UTC" deadline="2028-04-20 17:47:14.905358422 +0000 UTC"
Jul 10 18:58:39.329958 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.329898    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15598h48m35.575465302s"
Jul 10 18:58:39.354983 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.354942    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3179a044_43c4_48c9_9175_810563d50e49.slice/crio-6c1483484d807119ee1d884166c7ef168955a5078b3f8c8cbc38d938e72d8c50 WatchSource:0}: Error finding container 6c1483484d807119ee1d884166c7ef168955a5078b3f8c8cbc38d938e72d8c50: Status 404 returned error can't find the container with id 6c1483484d807119ee1d884166c7ef168955a5078b3f8c8cbc38d938e72d8c50
Jul 10 18:58:39.410013 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.409979    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:39.410518 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:39.410115    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:39.422709 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.422667    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsggr" event={"ID":"3179a044-43c4-48c9-9175-810563d50e49","Type":"ContainerStarted","Data":"6c1483484d807119ee1d884166c7ef168955a5078b3f8c8cbc38d938e72d8c50"}
Jul 10 18:58:39.424018 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.423985    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5rzb" event={"ID":"d6be89d1-7d75-49fd-b6fb-23a749d08caf","Type":"ContainerStarted","Data":"417e31728d16c1bb2db545c6fda2f9ee5d20156787af23e2759b9d4b8bf6e898"}
Jul 10 18:58:39.425338 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.425298    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nfh97" event={"ID":"8ab21d36-15af-4811-b62b-0acee70cee18","Type":"ContainerStarted","Data":"015a18888d120b8033eab9f1c93a96fd78ea00d36f2787cdba19821b2a133db0"}
Jul 10 18:58:39.426380 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.426352    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerStarted","Data":"9c9d863e968797a99abb88998eeb6d6c4af603a952945ba36e2b4c19941b01b8"}
Jul 10 18:58:39.794629 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.794592    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a57bc5d_8e88_42b6_8fdf_2f262c61125a.slice/crio-4b7c7439bce0a5a433392196c56abc050e1e8a9b9751fce237141c51c2e8f38f WatchSource:0}: Error finding container 4b7c7439bce0a5a433392196c56abc050e1e8a9b9751fce237141c51c2e8f38f: Status 404 returned error can't find the container with id 4b7c7439bce0a5a433392196c56abc050e1e8a9b9751fce237141c51c2e8f38f
Jul 10 18:58:39.796412 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.796375    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3783f4ff_7409_46f9_9c40_7beea2ada6c0.slice/crio-46417caf1b00736dd542040f0d7aa38f435dafd25e3e1db52fbebfc10d583efb WatchSource:0}: Error finding container 46417caf1b00736dd542040f0d7aa38f435dafd25e3e1db52fbebfc10d583efb: Status 404 returned error can't find the container with id 46417caf1b00736dd542040f0d7aa38f435dafd25e3e1db52fbebfc10d583efb
Jul 10 18:58:39.799605 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.799577    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55660e51_18bc_4cc7_a0d4_16036b2d9ec1.slice/crio-fb6b939e1096fb38ad875a3458e68f547c222b5ea22095e115ed19f6b55b25f5 WatchSource:0}: Error finding container fb6b939e1096fb38ad875a3458e68f547c222b5ea22095e115ed19f6b55b25f5: Status 404 returned error can't find the container with id fb6b939e1096fb38ad875a3458e68f547c222b5ea22095e115ed19f6b55b25f5
Jul 10 18:58:39.800634 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.800545    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d1484d_d443_4c20_ae86_5eeb82fc60b4.slice/crio-0f9dbd5fceb30932a32a7d7bdcc002935351cb624b9facf3aba419f7860a5780 WatchSource:0}: Error finding container 0f9dbd5fceb30932a32a7d7bdcc002935351cb624b9facf3aba419f7860a5780: Status 404 returned error can't find the container with id 0f9dbd5fceb30932a32a7d7bdcc002935351cb624b9facf3aba419f7860a5780
Jul 10 18:58:39.801669 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:58:39.801640    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1b3146_0f9e_49f0_977d_8887ff75c7a4.slice/crio-2bf214958e6e7eb71732a45ea391c71e4234eb1a943ee4484cd3bd6a5efc6c3c WatchSource:0}: Error finding container 2bf214958e6e7eb71732a45ea391c71e4234eb1a943ee4484cd3bd6a5efc6c3c: Status 404 returned error can't find the container with id 2bf214958e6e7eb71732a45ea391c71e4234eb1a943ee4484cd3bd6a5efc6c3c
Jul 10 18:58:39.812771 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:39.812742    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.007585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.007878    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.007939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.007961    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:42.0079385 +0000 UTC m=+6.141869151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.008026    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008047    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:40.008129 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008098    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:58:42.008082799 +0000 UTC m=+6.142013464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:40.008681 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008162    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:40.008681 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008194    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:40.008681 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008207    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:40.008681 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.008246    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:42.008233521 +0000 UTC m=+6.142164186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:40.331036 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.330949    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:39 +0000 UTC" deadline="2028-04-18 18:11:14.442442486 +0000 UTC"
Jul 10 18:58:40.331036 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.331027    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15551h12m34.111420727s"
Jul 10 18:58:40.409651 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.409572    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:40.409827 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.409718    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:40.410554 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.410531    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:40.410967 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:40.410642    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:40.430610 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.430570    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"2bf214958e6e7eb71732a45ea391c71e4234eb1a943ee4484cd3bd6a5efc6c3c"}
Jul 10 18:58:40.432397 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.432356    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcb4c" event={"ID":"3783f4ff-7409-46f9-9c40-7beea2ada6c0","Type":"ContainerStarted","Data":"46417caf1b00736dd542040f0d7aa38f435dafd25e3e1db52fbebfc10d583efb"}
Jul 10 18:58:40.435639 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.435580    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"b969a79c02bbbfd14d439b040b8bbd48","Type":"ContainerStarted","Data":"42b21401f54ec031b11c02b15fd0e3b21b2732f89fd5b4daf88fb20148f23b11"}
Jul 10 18:58:40.437444 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.437413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9496x" event={"ID":"c6d1484d-d443-4c20-ae86-5eeb82fc60b4","Type":"ContainerStarted","Data":"0f9dbd5fceb30932a32a7d7bdcc002935351cb624b9facf3aba419f7860a5780"}
Jul 10 18:58:40.438874 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.438848    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f" event={"ID":"55660e51-18bc-4cc7-a0d4-16036b2d9ec1","Type":"ContainerStarted","Data":"fb6b939e1096fb38ad875a3458e68f547c222b5ea22095e115ed19f6b55b25f5"}
Jul 10 18:58:40.441097 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.441073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hcf89" event={"ID":"2a57bc5d-8e88-42b6-8fdf-2f262c61125a","Type":"ContainerStarted","Data":"4b7c7439bce0a5a433392196c56abc050e1e8a9b9751fce237141c51c2e8f38f"}
Jul 10 18:58:40.442869 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:40.442844    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" event={"ID":"a274b38950da356e83a1d1b30ba640f1","Type":"ContainerStarted","Data":"04b5db33dbc04145a9b03a4921a4ad8ab10a15fdfa775538111a3824515d3962"}
Jul 10 18:58:41.410008 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.409411    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:41.410008 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:41.409574    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:41.459998 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.459954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5rzb" event={"ID":"d6be89d1-7d75-49fd-b6fb-23a749d08caf","Type":"ContainerStarted","Data":"4c25c3c56884d1647930177cca09122904f80184746fb9b3442dafaee8c65d44"}
Jul 10 18:58:41.473403 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.473361    2570 generic.go:356] "Generic (PLEG): container finished" podID="b969a79c02bbbfd14d439b040b8bbd48" containerID="42b21401f54ec031b11c02b15fd0e3b21b2732f89fd5b4daf88fb20148f23b11" exitCode=0
Jul 10 18:58:41.474912 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.474879    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"b969a79c02bbbfd14d439b040b8bbd48","Type":"ContainerDied","Data":"42b21401f54ec031b11c02b15fd0e3b21b2732f89fd5b4daf88fb20148f23b11"}
Jul 10 18:58:41.494034 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.493904    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g5rzb" podStartSLOduration=2.470518817 podStartE2EDuration="4.493881938s" totalImagesPullingTime="2.023363121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:41.492660735 +0000 UTC m=+5.626591410" watchObservedRunningTime="2026-07-10 18:58:41.493881938 +0000 UTC m=+5.627812609"
Jul 10 18:58:41.494553 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:41.494513    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" podStartSLOduration=3.494493003 podStartE2EDuration="3.494493003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:40.47460047 +0000 UTC m=+4.608531144" watchObservedRunningTime="2026-07-10 18:58:41.494493003 +0000 UTC m=+5.628423685"
Jul 10 18:58:42.030889 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:42.030844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:42.031143 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:42.030915    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:42.031143 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:42.030959    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:42.031143 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031117    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:42.031143 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031140    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:42.031452 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031153    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:42.031452 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031220    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.031199557 +0000 UTC m=+10.165130205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:42.031660 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031641    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:42.031735 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031699    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.031682797 +0000 UTC m=+10.165613448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:42.031796 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031755    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:42.031796 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.031787    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.031776835 +0000 UTC m=+10.165707487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:42.412301 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:42.412196    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:42.412482 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.412339    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:42.412787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:42.412765    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:42.412907 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:42.412887    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:43.409104 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:43.409064    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:43.409616 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:43.409219    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:43.908154 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:43.907949    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:43.908755 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:43.908735    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:44.408893 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:44.408858    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:44.408893 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:44.408885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:44.409139 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:44.408987    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:44.409504 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:44.409135    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:45.409756 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:45.409473    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:45.409756 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:45.409641    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:46.066944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:46.067011    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:46.067066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067073    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067160    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.067143906 +0000 UTC m=+18.201074553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067184    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067242    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.067221127 +0000 UTC m=+18.201151779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067324    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067342    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067354    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:46.067419 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.067388    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:54.067377474 +0000 UTC m=+18.201308141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:46.411495 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:46.410718    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:46.411495 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.410838    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:46.411495 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:46.411293    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:46.411495 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:46.411393    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:47.409225 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:47.409189    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:47.409447 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:47.409351    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:48.410164 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:48.410077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:48.410633 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:48.410227    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:48.411046 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:48.410973    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:48.411154 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:48.411135    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:49.409508 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:49.409471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:49.409678 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:49.409617    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:50.409953 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:50.409829    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:50.410385 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:50.409962    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:50.410385 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:50.410023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:50.410385 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:50.410158    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:51.409278 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:51.409244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:51.409452 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:51.409398    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:52.409669 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:52.409639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:52.410118 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:52.409639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:52.410118 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:52.409751    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:52.410118 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:52.409844    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:53.409974 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:53.409933    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:53.410442 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:53.410078    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:54.129806 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:54.129705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:54.129806 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:54.129786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:54.129837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.129900    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.129924    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.129971    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.129952396 +0000 UTC m=+34.263883054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.129980    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.130000    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:58:54.130034 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.130014    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:54.130278 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.129985    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.129978945 +0000 UTC m=+34.263909590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:58:54.130278 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.130070    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:10.130057189 +0000 UTC m=+34.263987835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:58:54.409532 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:54.409458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:54.409667 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.409594    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:54.409667 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:54.409634    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:54.409754 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:54.409709    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:55.409569 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:55.409534    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:55.409955 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:55.409681    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:56.116262 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:56.116230    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:56.116430 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:56.116362    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 10 18:58:56.116842 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:56.116816    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g5rzb"
Jul 10 18:58:56.409887 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:56.409812    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:56.410259 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:56.409903    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:56.410259 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:56.409991    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:56.410259 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:56.410102    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:57.409638 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:57.409601    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:57.409814 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:57.409745    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:58.412271 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.412091    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:58:58.412885 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.412119    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:58:58.412885 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:58.412373    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:58:58.412885 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:58.412458    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:58:58.514676 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.514594    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hcf89" event={"ID":"2a57bc5d-8e88-42b6-8fdf-2f262c61125a","Type":"ContainerStarted","Data":"9f4b70d160b2597ac340fd480f529da4fb0227ae878423188e1d60ed4988b7d4"}
Jul 10 18:58:58.516054 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.516030    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsggr" event={"ID":"3179a044-43c4-48c9-9175-810563d50e49","Type":"ContainerStarted","Data":"2716bfdac41eaa3692ad75cc32e38b09447d6968ef4722932fac136ed2336a74"}
Jul 10 18:58:58.517528 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.517503    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nfh97" event={"ID":"8ab21d36-15af-4811-b62b-0acee70cee18","Type":"ContainerStarted","Data":"37974a640234ab569b444d1af48dd2af05fdc098489cc7b9a598c4c6ef8fdf16"}
Jul 10 18:58:58.520691 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.520669    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"b969a79c02bbbfd14d439b040b8bbd48","Type":"ContainerStarted","Data":"062c6c4048ddb4ab0468e7e66c8ba740939e718502c86722f54e5c54242229c3"}
Jul 10 18:58:58.588134 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.588069    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hcf89" podStartSLOduration=3.23519551 podStartE2EDuration="21.58804557s" totalImagesPullingTime="18.35285006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:58.547887374 +0000 UTC m=+22.681818042" watchObservedRunningTime="2026-07-10 18:58:58.58804557 +0000 UTC m=+22.721976238"
Jul 10 18:58:58.625117 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.625063    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" podStartSLOduration=20.625047192 podStartE2EDuration="20.625047192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:58.624897584 +0000 UTC m=+22.758828250" watchObservedRunningTime="2026-07-10 18:58:58.625047192 +0000 UTC m=+22.758977858"
Jul 10 18:58:58.625313 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.625293    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xsggr" podStartSLOduration=2.842113213 podStartE2EDuration="21.625286888s" totalImagesPullingTime="18.783173675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:58.589475618 +0000 UTC m=+22.723406285" watchObservedRunningTime="2026-07-10 18:58:58.625286888 +0000 UTC m=+22.759217555"
Jul 10 18:58:58.659695 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:58.659654    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nfh97" podStartSLOduration=2.161346927 podStartE2EDuration="21.659639663s" totalImagesPullingTime="19.498292736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:58.659543912 +0000 UTC m=+22.793474569" watchObservedRunningTime="2026-07-10 18:58:58.659639663 +0000 UTC m=+22.793570329"
Jul 10 18:58:59.409959 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.409686    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:58:59.410234 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:58:59.410022    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:58:59.524209 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.524175    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="11beade1a4717d26c4f0a0f64497a6ca463070ff4f28bf6f6927a2d28928f9cd" exitCode=0
Jul 10 18:58:59.524878 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.524265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"11beade1a4717d26c4f0a0f64497a6ca463070ff4f28bf6f6927a2d28928f9cd"}
Jul 10 18:58:59.527262 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527246    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 18:58:59.527593 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527571    2570 generic.go:356] "Generic (PLEG): container finished" podID="9a1b3146-0f9e-49f0-977d-8887ff75c7a4" containerID="79b3cb6be941157aa4e9ad1328a0313b7bfa9900df84a250db779a2af5435ceb" exitCode=1
Jul 10 18:58:59.527668 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527645    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"1f1ffd65d55f3d8d2b0e7863f47b430da1ef469985c892cbeaa3a17c6acdbfeb"}
Jul 10 18:58:59.527750 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527733    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"99c33f601a3a5d421b0e8b04e5fc666fccc4348e6f215a402fab6cf706829db0"}
Jul 10 18:58:59.527975 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527795    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"ce7aff6ed1be2f5446026e5cfe5da5ff0035e99d2b38c742d5508201233fad32"}
Jul 10 18:58:59.527975 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527828    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"39dc9feb1c0d53580271e2a5660803acdcfde0146e3e2483fbf7100f67fdd6ac"}
Jul 10 18:58:59.527975 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527857    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"1208060a1e3d3e4032f6c084753aee1ca812ed7fb3f8c0528109e78176e084b9"}
Jul 10 18:58:59.527975 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.527903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerDied","Data":"79b3cb6be941157aa4e9ad1328a0313b7bfa9900df84a250db779a2af5435ceb"}
Jul 10 18:58:59.529093 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.529068    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcb4c" event={"ID":"3783f4ff-7409-46f9-9c40-7beea2ada6c0","Type":"ContainerStarted","Data":"0d303c55689249de0dd6015b31d4303807cdfaec746b8ec52329351606deff23"}
Jul 10 18:58:59.530668 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.530642    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9496x" event={"ID":"c6d1484d-d443-4c20-ae86-5eeb82fc60b4","Type":"ContainerStarted","Data":"20b1e19a71724181b20c55b5ce94a63c3389a358f2a173d80fbadd032cb81b32"}
Jul 10 18:58:59.532096 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.532072    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f" event={"ID":"55660e51-18bc-4cc7-a0d4-16036b2d9ec1","Type":"ContainerStarted","Data":"9b8c7bc631e83f281593061f1956a534bffb36a875a6d7afdbe8d19dc15016d2"}
Jul 10 18:58:59.587621 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.587571    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9496x" podStartSLOduration=4.24711031 podStartE2EDuration="22.587555245s" totalImagesPullingTime="18.340444935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.587472353 +0000 UTC m=+23.721403019" watchObservedRunningTime="2026-07-10 18:58:59.587555245 +0000 UTC m=+23.721485911"
Jul 10 18:58:59.911376 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:58:59.911350    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:59:00.358215 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.358065    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:58:59.911369746Z","UUID":"e2586c41-9f27-45ee-98e8-e2d368436466","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:59:00.359917 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.359891    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:59:00.360058 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.359937    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:59:00.409489 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.409449    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:00.409666 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:00.409596    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:00.409817 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.409449    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:00.409916 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:00.409895    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:00.536842 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:00.536807    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f" event={"ID":"55660e51-18bc-4cc7-a0d4-16036b2d9ec1","Type":"ContainerStarted","Data":"8cd21fa014f2c1e348130f87e8a7e4c20b6f2cd0c356d9b830ad1681ea5d965e"}
Jul 10 18:59:01.409990 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:01.409902    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:01.410137 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:01.410019    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:01.542675 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:01.542644    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 18:59:01.543165 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:01.543133    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"6cae82706f091a8e52e06e1c09271f193a0be4f5d7ea98d0ba153439a0414ece"}
Jul 10 18:59:01.545403 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:01.545339    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f" event={"ID":"55660e51-18bc-4cc7-a0d4-16036b2d9ec1","Type":"ContainerStarted","Data":"3d72cc826f8e72840e1d5488c6bddc423044126571d0c4aaefcdb9a02edfe8a3"}
Jul 10 18:59:01.565519 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:01.565450    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kcb4c" podStartSLOduration=6.223465777 podStartE2EDuration="24.565377008s" totalImagesPullingTime="18.341911231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.623888305 +0000 UTC m=+23.757818971" watchObservedRunningTime="2026-07-10 18:59:01.565377008 +0000 UTC m=+25.699307676"
Jul 10 18:59:02.409945 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:02.409741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:02.410119 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:02.409817    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:02.410119 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:02.410048    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:02.410119 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:02.410111    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:03.409707 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:03.409663    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:03.410113 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:03.409800    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:04.410104 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.409926    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:04.410849 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.409926    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:04.410849 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:04.410179    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:04.410849 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:04.410269    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:04.553537 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.553488    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="7ebfc8be864182b5aab41070ef2d3447ea97037bf64bb4e32ed6603e676f3a1c" exitCode=0
Jul 10 18:59:04.553714 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.553557    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"7ebfc8be864182b5aab41070ef2d3447ea97037bf64bb4e32ed6603e676f3a1c"}
Jul 10 18:59:04.557108 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.557090    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 18:59:04.557431 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.557413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"86ac6781d794089483d6eeb0cbe50e8e94bce98bc48343a3b632965f3e7ffcc4"}
Jul 10 18:59:04.557674 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.557661    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:04.557717 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.557711    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:04.557855 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.557839    2570 scope.go:117] "RemoveContainer" containerID="79b3cb6be941157aa4e9ad1328a0313b7bfa9900df84a250db779a2af5435ceb"
Jul 10 18:59:04.577203 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.577153    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l284f" podStartSLOduration=7.139760077 podStartE2EDuration="27.577132551s" totalImagesPullingTime="20.437372474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:01.566040608 +0000 UTC m=+25.699971274" watchObservedRunningTime="2026-07-10 18:59:04.577132551 +0000 UTC m=+28.711063219"
Jul 10 18:59:04.578477 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:04.578457    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:05.409655 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.409625    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:05.409865 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:05.409737    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:05.564892 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.564536    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 18:59:05.565146 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.564988    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" event={"ID":"9a1b3146-0f9e-49f0-977d-8887ff75c7a4","Type":"ContainerStarted","Data":"d61dbb38174377902090f34d197acda16b8d91322cd926cea935db2b5afd60a2"}
Jul 10 18:59:05.565190 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.565166    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:05.585667 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.585236    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:05.594345 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.594292    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx" podStartSLOduration=10.133742526 podStartE2EDuration="28.594278395s" totalImagesPullingTime="18.460535869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:05.594054387 +0000 UTC m=+29.727985054" watchObservedRunningTime="2026-07-10 18:59:05.594278395 +0000 UTC m=+29.728209061"
Jul 10 18:59:05.783927 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.783886    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8qdk8"]
Jul 10 18:59:05.784088 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.783990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:05.784088 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:05.784079    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:05.787705 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.787674    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jg5tw"]
Jul 10 18:59:05.787868 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.787808    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:05.787931 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:05.787895    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:05.788350 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.788327    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6d6ql"]
Jul 10 18:59:05.788462 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:05.788418    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:05.788517 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:05.788485    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:06.569162 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:06.569123    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="0faf45a369aa0ca736fbb7436463c5b203c43264e431982bcf6bda6859557eee" exitCode=0
Jul 10 18:59:06.569538 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:06.569211    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"0faf45a369aa0ca736fbb7436463c5b203c43264e431982bcf6bda6859557eee"}
Jul 10 18:59:07.410019 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:07.409860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:07.410180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:07.409899    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:07.410180 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:07.410112    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:07.410180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:07.409932    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:07.410290 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:07.410158    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:07.410290 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:07.410244    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:08.575861 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:08.575820    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="a3e37e1eb434190de89e5f8dfc13eedf1eeb8763f19cd043f84f5d0b4c075162" exitCode=0
Jul 10 18:59:08.576462 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:08.575873    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"a3e37e1eb434190de89e5f8dfc13eedf1eeb8763f19cd043f84f5d0b4c075162"}
Jul 10 18:59:08.789269 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:08.789233    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g5rzb_d6be89d1-7d75-49fd-b6fb-23a749d08caf/konnectivity-agent/0.log" limit=1024
Jul 10 18:59:09.409532 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:09.409496    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:09.409702 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:09.409539    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:09.409702 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:09.409496    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:09.409702 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:09.409625    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jg5tw" podUID="07e43abd-974e-4a92-9898-e886edee0be1"
Jul 10 18:59:09.409842 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:09.409710    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6d6ql" podUID="17b396ae-7bc9-41ef-8a1c-5a2e10de647a"
Jul 10 18:59:09.409842 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:09.409797    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qdk8" podUID="a0f255b1-7837-4251-9bca-3aa6563d4333"
Jul 10 18:59:10.150579 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:10.150548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:10.150590    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:10.150619    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150736    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150746    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150760    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150788    2570 projected.go:196] Error preparing data for projected volume kube-api-access-m7fxb for pod openshift-network-diagnostics/network-check-target-jg5tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150813    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret podName:17b396ae-7bc9-41ef-8a1c-5a2e10de647a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:42.150797852 +0000 UTC m=+66.284728499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret") pod "global-pull-secret-syncer-6d6ql" (UID: "17b396ae-7bc9-41ef-8a1c-5a2e10de647a") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150820    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150835    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb podName:07e43abd-974e-4a92-9898-e886edee0be1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:42.150821367 +0000 UTC m=+66.284752027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fxb" (UniqueName: "kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb") pod "network-check-target-jg5tw" (UID: "07e43abd-974e-4a92-9898-e886edee0be1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:59:10.151042 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:10.150862    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:42.150849572 +0000 UTC m=+66.284780227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:59:11.158317 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.158289    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeReady"
Jul 10 18:59:11.158887 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.158426    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:59:11.211909 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.211875    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mllg9"]
Jul 10 18:59:11.217075 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.217046    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.218051 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.218021    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-km6xx"]
Jul 10 18:59:11.219975 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.219948    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:59:11.220099 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.220044    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4zmd\""
Jul 10 18:59:11.220173 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.220141    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:59:11.221264 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.221246    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.224836 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.224816    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:59:11.225251 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.224934    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:59:11.225251 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.225100    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xqmcj\""
Jul 10 18:59:11.225447 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.225424    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:59:11.235137 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.235113    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mllg9"]
Jul 10 18:59:11.235875 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.235840    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-km6xx"]
Jul 10 18:59:11.358494 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-config-volume\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.358494 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-tmp-dir\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.358724 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358545    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-cert\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.358724 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gs9l\" (UniqueName: \"kubernetes.io/projected/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-kube-api-access-8gs9l\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.358724 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358654    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.358724 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.358690    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtmf\" (UniqueName: \"kubernetes.io/projected/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-kube-api-access-hmtmf\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.409878 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.409795    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:11.409878 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.409812    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:11.409878 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.409812    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:11.413217 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.413192    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:59:11.413367 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.413231    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:59:11.413508 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.413494    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:59:11.413508 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.413502    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:59:11.413918 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.413682    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66xb5\""
Jul 10 18:59:11.414071 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.414051    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvrkc\""
Jul 10 18:59:11.459944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.459893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.459944 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.459947    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtmf\" (UniqueName: \"kubernetes.io/projected/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-kube-api-access-hmtmf\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.460192 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-config-volume\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.460192 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460030    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-tmp-dir\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.460192 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:11.460083    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:11.460192 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:11.460168    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:11.960145821 +0000 UTC m=+36.094076476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:11.460397 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460094    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-cert\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.460397 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460267    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gs9l\" (UniqueName: \"kubernetes.io/projected/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-kube-api-access-8gs9l\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.460512 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460489    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-tmp-dir\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.460948 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.460925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-config-volume\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.464688 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.464668    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-cert\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.471301 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.471274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gs9l\" (UniqueName: \"kubernetes.io/projected/fdca6c0c-efeb-4cae-876c-e9e55d0c24a3-kube-api-access-8gs9l\") pod \"ingress-canary-km6xx\" (UID: \"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3\") " pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.471622 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.471600    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtmf\" (UniqueName: \"kubernetes.io/projected/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-kube-api-access-hmtmf\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.535194 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.535157    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-km6xx"
Jul 10 18:59:11.964396 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:11.964358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:11.964574 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:11.964545    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:11.964633 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:11.964619    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:12.964597134 +0000 UTC m=+37.098527780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:12.500583 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:12.500344    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-km6xx"]
Jul 10 18:59:12.502841 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:12.502798    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdca6c0c_efeb_4cae_876c_e9e55d0c24a3.slice/crio-1b3ebd2729a4f89151afa3b7dd89df80f1de5e1acdbbdf61403509e17134f255 WatchSource:0}: Error finding container 1b3ebd2729a4f89151afa3b7dd89df80f1de5e1acdbbdf61403509e17134f255: Status 404 returned error can't find the container with id 1b3ebd2729a4f89151afa3b7dd89df80f1de5e1acdbbdf61403509e17134f255
Jul 10 18:59:12.586967 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:12.586930    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-km6xx" event={"ID":"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3","Type":"ContainerStarted","Data":"1b3ebd2729a4f89151afa3b7dd89df80f1de5e1acdbbdf61403509e17134f255"}
Jul 10 18:59:12.589684 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:12.589642    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerStarted","Data":"ff9c10d2d3e0cdae8eb34fce43ba0c85af695d63b5c814b2b3d030f6301ae92d"}
Jul 10 18:59:12.971537 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:12.971502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:12.971732 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:12.971659    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:12.971831 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:12.971741    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:14.971717767 +0000 UTC m=+39.105648413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:13.594903 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:13.594871    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="ff9c10d2d3e0cdae8eb34fce43ba0c85af695d63b5c814b2b3d030f6301ae92d" exitCode=0
Jul 10 18:59:13.595351 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:13.594935    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"ff9c10d2d3e0cdae8eb34fce43ba0c85af695d63b5c814b2b3d030f6301ae92d"}
Jul 10 18:59:14.599081 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:14.599047    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-km6xx" event={"ID":"fdca6c0c-efeb-4cae-876c-e9e55d0c24a3","Type":"ContainerStarted","Data":"c4a58012e21af2f71a69edaa85f07157b96cc78222e13d357fab1797917f4456"}
Jul 10 18:59:14.601745 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:14.601719    2570 generic.go:356] "Generic (PLEG): container finished" podID="b226814b-8a33-46dc-af68-bafeb83fa909" containerID="801f83648e6454678688d2ef73eef258b8d945d05a088ce44b2cd8cc64759ddd" exitCode=0
Jul 10 18:59:14.601855 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:14.601785    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerDied","Data":"801f83648e6454678688d2ef73eef258b8d945d05a088ce44b2cd8cc64759ddd"}
Jul 10 18:59:14.987733 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:14.987461    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:14.987929 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:14.987625    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:14.987929 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:14.987845    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:18.987825693 +0000 UTC m=+43.121756342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:15.607167 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:15.607089    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" event={"ID":"b226814b-8a33-46dc-af68-bafeb83fa909","Type":"ContainerStarted","Data":"3f5f57e80cae1f54afca7f01b69151cf94d3ed13fb460fa103e59b3443df5b0a"}
Jul 10 18:59:15.627835 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:15.627786    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-km6xx" podStartSLOduration=2.645148969 podStartE2EDuration="4.627752509s" totalImagesPullingTime="1.98260354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:15.627546167 +0000 UTC m=+39.761476834" watchObservedRunningTime="2026-07-10 18:59:15.627752509 +0000 UTC m=+39.761683175"
Jul 10 18:59:15.654836 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:15.654785    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m2xd5" podStartSLOduration=9.104415637 podStartE2EDuration="38.654753219s" totalImagesPullingTime="29.550337582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:15.65322345 +0000 UTC m=+39.787154123" watchObservedRunningTime="2026-07-10 18:59:15.654753219 +0000 UTC m=+39.788683886"
Jul 10 18:59:19.014971 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:19.014921    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:19.015390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:19.015084    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:19.015390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:19.015152    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:27.01513499 +0000 UTC m=+51.149065635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:27.075873 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:27.075828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:27.076348 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:27.075942    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:27.076348 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:27.076003    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:43.075990513 +0000 UTC m=+67.209921159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:37.581626 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:37.581598    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5mrx"
Jul 10 18:59:39.874093 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.874057    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-jhgc6"]
Jul 10 18:59:39.876750 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.876734    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.879213 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.879185    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 10 18:59:39.879213 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.879201    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 10 18:59:39.879378 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.879199    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:59:39.879378 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.879326    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cxgtb\""
Jul 10 18:59:39.880132 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.880114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:59:39.883940 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.883923    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 10 18:59:39.889222 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.889200    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-jhgc6"]
Jul 10 18:59:39.965540 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.965540 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/256d0b26-9b34-46d7-9b6e-e05911b13b66-serving-cert\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.965787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-snapshots\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.965787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965672    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.965787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx66s\" (UniqueName: \"kubernetes.io/projected/256d0b26-9b34-46d7-9b6e-e05911b13b66-kube-api-access-xx66s\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:39.965787 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:39.965732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-tmp\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066521 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-tmp\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066716 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066716 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/256d0b26-9b34-46d7-9b6e-e05911b13b66-serving-cert\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066716 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066571    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-snapshots\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066716 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066621    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.066716 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx66s\" (UniqueName: \"kubernetes.io/projected/256d0b26-9b34-46d7-9b6e-e05911b13b66-kube-api-access-xx66s\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.067004 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.066974    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-tmp\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.067200 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.067179    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.067261 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.067197    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/256d0b26-9b34-46d7-9b6e-e05911b13b66-snapshots\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.067457 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.067440    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256d0b26-9b34-46d7-9b6e-e05911b13b66-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.070427 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.070404    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/256d0b26-9b34-46d7-9b6e-e05911b13b66-serving-cert\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.081304 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.081273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx66s\" (UniqueName: \"kubernetes.io/projected/256d0b26-9b34-46d7-9b6e-e05911b13b66-kube-api-access-xx66s\") pod \"insights-operator-68d9dd94cf-jhgc6\" (UID: \"256d0b26-9b34-46d7-9b6e-e05911b13b66\") " pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.176585 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.176554    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-648566b76d-5cwnc"]
Jul 10 18:59:40.178731 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.178713    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.181886 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.181866    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 10 18:59:40.181988 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.181961    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 10 18:59:40.182092 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.182055    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 10 18:59:40.182169 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.182157    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 10 18:59:40.182379 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.182365    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 10 18:59:40.182432 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.182375    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jfcq4\""
Jul 10 18:59:40.182588 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.182572    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 10 18:59:40.185130 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.185103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6"
Jul 10 18:59:40.200174 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.200135    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648566b76d-5cwnc"]
Jul 10 18:59:40.268266 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.268233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.268413 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.268342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48xl\" (UniqueName: \"kubernetes.io/projected/846ea7b0-fbdc-4a3c-93d7-be041d80210a-kube-api-access-f48xl\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.268413 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.268391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-stats-auth\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.268518 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.268453    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.268575 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.268525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-default-certificate\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.313997 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.313958    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-jhgc6"]
Jul 10 18:59:40.317019 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:40.316986    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256d0b26_9b34_46d7_9b6e_e05911b13b66.slice/crio-cb8c1f651495eef8678fb50adc0ab5cc19f15d8b57f60cc31a365165cccbb078 WatchSource:0}: Error finding container cb8c1f651495eef8678fb50adc0ab5cc19f15d8b57f60cc31a365165cccbb078: Status 404 returned error can't find the container with id cb8c1f651495eef8678fb50adc0ab5cc19f15d8b57f60cc31a365165cccbb078
Jul 10 18:59:40.369608 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.369560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-stats-auth\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.369798 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.369632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.369798 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.369661    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-default-certificate\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.369798 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.369727    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.369798 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.369785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48xl\" (UniqueName: \"kubernetes.io/projected/846ea7b0-fbdc-4a3c-93d7-be041d80210a-kube-api-access-f48xl\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.370076 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.369828    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:40.869809024 +0000 UTC m=+65.003739691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:40.370076 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.369879    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:40.370076 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.369945    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:40.869927971 +0000 UTC m=+65.003858616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 18:59:40.372080 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.372062    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-default-certificate\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.372157 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.372091    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-stats-auth\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.378660 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.378641    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48xl\" (UniqueName: \"kubernetes.io/projected/846ea7b0-fbdc-4a3c-93d7-be041d80210a-kube-api-access-f48xl\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.664077 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.664042    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6" event={"ID":"256d0b26-9b34-46d7-9b6e-e05911b13b66","Type":"ContainerStarted","Data":"cb8c1f651495eef8678fb50adc0ab5cc19f15d8b57f60cc31a365165cccbb078"}
Jul 10 18:59:40.873503 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.873470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.873701 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:40.873542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:40.873701 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.873636    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:40.873701 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.873669    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.873646717 +0000 UTC m=+66.007577382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:40.873701 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:40.873689    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.873680897 +0000 UTC m=+66.007611551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 18:59:41.882841 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:41.882802    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:41.883257 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:41.882965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:41.883257 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:41.883000    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:43.882979461 +0000 UTC m=+68.016910110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:41.883257 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:41.883079    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:41.883257 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:41.883129    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:43.883115964 +0000 UTC m=+68.017046614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 18:59:42.184743 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.184706    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 18:59:42.184954 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.184784    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:42.184954 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.184813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:42.187538 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.187517    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:59:42.187640 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.187549    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:59:42.187640 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.187554    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:59:42.195868 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:42.195838    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 18:59:42.196046 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:42.195932    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs podName:a0f255b1-7837-4251-9bca-3aa6563d4333 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:46.19591415 +0000 UTC m=+130.329844796 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs") pod "network-metrics-daemon-8qdk8" (UID: "a0f255b1-7837-4251-9bca-3aa6563d4333") : secret "metrics-daemon-secret" not found
Jul 10 18:59:42.198016 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.197992    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17b396ae-7bc9-41ef-8a1c-5a2e10de647a-original-pull-secret\") pod \"global-pull-secret-syncer-6d6ql\" (UID: \"17b396ae-7bc9-41ef-8a1c-5a2e10de647a\") " pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:42.198521 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.198507    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:59:42.208617 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.208592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fxb\" (UniqueName: \"kubernetes.io/projected/07e43abd-974e-4a92-9898-e886edee0be1-kube-api-access-m7fxb\") pod \"network-check-target-jg5tw\" (UID: \"07e43abd-974e-4a92-9898-e886edee0be1\") " pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:42.326755 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.326722    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66xb5\""
Jul 10 18:59:42.333986 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.333963    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:42.335905 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.335880    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6d6ql"
Jul 10 18:59:42.464899 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.464824    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6d6ql"]
Jul 10 18:59:42.468363 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:42.468341    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b396ae_7bc9_41ef_8a1c_5a2e10de647a.slice/crio-f2889ce711bfe15ab9368e624842ca275ba1e5560af03aa48a9f2e5aee8e35e6 WatchSource:0}: Error finding container f2889ce711bfe15ab9368e624842ca275ba1e5560af03aa48a9f2e5aee8e35e6: Status 404 returned error can't find the container with id f2889ce711bfe15ab9368e624842ca275ba1e5560af03aa48a9f2e5aee8e35e6
Jul 10 18:59:42.479180 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.479113    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jg5tw"]
Jul 10 18:59:42.482473 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:42.482451    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e43abd_974e_4a92_9898_e886edee0be1.slice/crio-f4c68b92a34b74bdbb071bac7ab6844e82375de822dbc1a00470bac6d7d6e074 WatchSource:0}: Error finding container f4c68b92a34b74bdbb071bac7ab6844e82375de822dbc1a00470bac6d7d6e074: Status 404 returned error can't find the container with id f4c68b92a34b74bdbb071bac7ab6844e82375de822dbc1a00470bac6d7d6e074
Jul 10 18:59:42.670616 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.670575    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6" event={"ID":"256d0b26-9b34-46d7-9b6e-e05911b13b66","Type":"ContainerStarted","Data":"2eb95a542bf05f02397253b6ef41174e61711b3a6b949370219a574904433205"}
Jul 10 18:59:42.671736 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.671712    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jg5tw" event={"ID":"07e43abd-974e-4a92-9898-e886edee0be1","Type":"ContainerStarted","Data":"f4c68b92a34b74bdbb071bac7ab6844e82375de822dbc1a00470bac6d7d6e074"}
Jul 10 18:59:42.672707 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.672685    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6d6ql" event={"ID":"17b396ae-7bc9-41ef-8a1c-5a2e10de647a","Type":"ContainerStarted","Data":"f2889ce711bfe15ab9368e624842ca275ba1e5560af03aa48a9f2e5aee8e35e6"}
Jul 10 18:59:42.688872 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:42.688823    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6" podStartSLOduration=1.806441763 podStartE2EDuration="3.688806926s" totalImagesPullingTime="1.882365163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:42.68774335 +0000 UTC m=+66.821674016" watchObservedRunningTime="2026-07-10 18:59:42.688806926 +0000 UTC m=+66.822737571"
Jul 10 18:59:43.091285 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:43.091243    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 18:59:43.091822 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:43.091467    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:59:43.091822 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:43.091541    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls podName:4d9bdc1d-3a17-4970-992f-fc348fd4a8e1 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:15.091522186 +0000 UTC m=+99.225452859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls") pod "dns-default-mllg9" (UID: "4d9bdc1d-3a17-4970-992f-fc348fd4a8e1") : secret "dns-default-metrics-tls" not found
Jul 10 18:59:43.897816 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:43.897756    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:43.898077 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:43.897868    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:43.898077 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:43.897895    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.897874622 +0000 UTC m=+72.031805291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:43.898077 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:43.897947    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:43.898077 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:43.897986    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.897973475 +0000 UTC m=+72.031904132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 18:59:44.566315 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.566056    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-blkm7"]
Jul 10 18:59:44.591283 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.590537    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.595858 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.595164    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:59:44.595858 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.595483    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:59:44.595858 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.595688    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pjpmw\""
Jul 10 18:59:44.596349 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.596320    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-blkm7"]
Jul 10 18:59:44.704214 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.704175    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc1193ca-6386-46ad-9f2f-0acc30d02996-crio-socket\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.704214 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.704220    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.704456 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.704329    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.704456 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.704376    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc1193ca-6386-46ad-9f2f-0acc30d02996-data-volume\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.704572 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.704501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqkt\" (UniqueName: \"kubernetes.io/projected/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-api-access-8jqkt\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805028 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.804983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805228 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805038    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc1193ca-6386-46ad-9f2f-0acc30d02996-data-volume\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805228 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqkt\" (UniqueName: \"kubernetes.io/projected/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-api-access-8jqkt\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805228 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:44.805165    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:44.805387 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:44.805242    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.305222328 +0000 UTC m=+69.439152988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:44.805387 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805332    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc1193ca-6386-46ad-9f2f-0acc30d02996-crio-socket\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805491 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc1193ca-6386-46ad-9f2f-0acc30d02996-crio-socket\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805550 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805493    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805550 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.805496    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc1193ca-6386-46ad-9f2f-0acc30d02996-data-volume\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:44.805646 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:44.805557    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:44.805646 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:44.805620    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.30560343 +0000 UTC m=+69.439534077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:44.814847 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:44.814820    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqkt\" (UniqueName: \"kubernetes.io/projected/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-api-access-8jqkt\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:45.310804 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.310748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:45.311000 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.310825    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:45.311000 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:45.310863    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:45.311000 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:45.310946    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:46.310925697 +0000 UTC m=+70.444856365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:45.311000 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:45.310963    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:45.311194 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:45.311040    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:46.311024386 +0000 UTC m=+70.444955034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:45.403655 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.403616    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"]
Jul 10 18:59:45.427634 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.427597    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"]
Jul 10 18:59:45.427804 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.427690    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.438178 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.438144    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 10 18:59:45.438178 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.438158    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:59:45.438369 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.438165    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:59:45.439588 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.439567    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:59:45.512799 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.512745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71457184-530b-4062-9819-7b5d418b8d37-klusterlet-config\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.512994 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.512965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljtp\" (UniqueName: \"kubernetes.io/projected/71457184-530b-4062-9819-7b5d418b8d37-kube-api-access-rljtp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.513058 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.513044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71457184-530b-4062-9819-7b5d418b8d37-tmp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.613599 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.613520    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljtp\" (UniqueName: \"kubernetes.io/projected/71457184-530b-4062-9819-7b5d418b8d37-kube-api-access-rljtp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.613599 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.613582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71457184-530b-4062-9819-7b5d418b8d37-tmp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.614024 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.613605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71457184-530b-4062-9819-7b5d418b8d37-klusterlet-config\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.614024 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.614001    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71457184-530b-4062-9819-7b5d418b8d37-tmp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.616027 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.616008    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71457184-530b-4062-9819-7b5d418b8d37-klusterlet-config\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.622543 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.622513    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljtp\" (UniqueName: \"kubernetes.io/projected/71457184-530b-4062-9819-7b5d418b8d37-kube-api-access-rljtp\") pod \"klusterlet-addon-workmgr-78c78c884f-lhls8\" (UID: \"71457184-530b-4062-9819-7b5d418b8d37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:45.738877 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:45.738831    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:46.318972 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:46.318938    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:46.319161 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:46.318988    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:46.319161 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:46.319063    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:46.319161 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:46.319149    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:48.319125503 +0000 UTC m=+72.453056152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:46.319300 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:46.319163    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:46.319300 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:46.319213    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:48.319198097 +0000 UTC m=+72.453128759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:46.698480 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:46.698451    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsggr_3179a044-43c4-48c9-9175-810563d50e49/dns-node-resolver/0.log"
Jul 10 18:59:47.133433 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.133408    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"]
Jul 10 18:59:47.138675 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:47.138647    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71457184_530b_4062_9819_7b5d418b8d37.slice/crio-85c722c7a5611fe4fd221b0adf7375a34b3c37506c8ac3db2b039dc0806e9831 WatchSource:0}: Error finding container 85c722c7a5611fe4fd221b0adf7375a34b3c37506c8ac3db2b039dc0806e9831: Status 404 returned error can't find the container with id 85c722c7a5611fe4fd221b0adf7375a34b3c37506c8ac3db2b039dc0806e9831
Jul 10 18:59:47.504791 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.504726    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcb4c_3783f4ff-7409-46f9-9c40-7beea2ada6c0/node-ca/0.log"
Jul 10 18:59:47.689111 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.688674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6d6ql" event={"ID":"17b396ae-7bc9-41ef-8a1c-5a2e10de647a","Type":"ContainerStarted","Data":"8239621e358cd3b017d3f3b17cbbe7bf0e9303bbc67408554c3fd89a337fa178"}
Jul 10 18:59:47.690182 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.690154    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8" event={"ID":"71457184-530b-4062-9819-7b5d418b8d37","Type":"ContainerStarted","Data":"85c722c7a5611fe4fd221b0adf7375a34b3c37506c8ac3db2b039dc0806e9831"}
Jul 10 18:59:47.692090 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.692035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jg5tw" event={"ID":"07e43abd-974e-4a92-9898-e886edee0be1","Type":"ContainerStarted","Data":"16bd00e20c360b4b59ec3d984c610fab60138608ff1e603478198b9df8f6b033"}
Jul 10 18:59:47.692229 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.692193    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 18:59:47.731554 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.730127    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6d6ql" podStartSLOduration=66.183997266 podStartE2EDuration="1m10.730105931s" totalImagesPullingTime="4.546108665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.711916494 +0000 UTC m=+71.845847162" watchObservedRunningTime="2026-07-10 18:59:47.730105931 +0000 UTC m=+71.864036604"
Jul 10 18:59:47.932059 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.932019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:47.932247 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:47.932132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:47.932330 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:47.932270    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:47.932330 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:47.932297    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:55.932276724 +0000 UTC m=+80.066207391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:47.932437 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:47.932337    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:55.932319257 +0000 UTC m=+80.066249914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 18:59:48.335159 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.335063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:48.335159 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.335132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:48.335390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:48.335228    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:48.335390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:48.335269    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:48.335390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:48.335304    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:52.335282333 +0000 UTC m=+76.469212998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:48.335390 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:48.335321    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:52.335313266 +0000 UTC m=+76.469243913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:48.518698 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.518656    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-km6xx_fdca6c0c-efeb-4cae-876c-e9e55d0c24a3/serve-healthcheck-canary/0.log"
Jul 10 18:59:48.554327 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.554266    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jg5tw" podStartSLOduration=67.031855453 podStartE2EDuration="1m11.554250036s" totalImagesPullingTime="4.522394583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.731524312 +0000 UTC m=+71.865454980" watchObservedRunningTime="2026-07-10 18:59:48.554250036 +0000 UTC m=+72.688180721"
Jul 10 18:59:48.555560 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.555525    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"]
Jul 10 18:59:48.561897 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.561874    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"
Jul 10 18:59:48.567018 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.566996    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 10 18:59:48.567137 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.567061    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xlrqj\""
Jul 10 18:59:48.567137 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.566996    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 10 18:59:48.584691 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.584662    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"]
Jul 10 18:59:48.638070 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.638041    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n52wq\" (UniqueName: \"kubernetes.io/projected/ce49190c-cda3-4a90-a536-ec1a35e98730-kube-api-access-n52wq\") pod \"migrator-7f59b54f6c-n5rrx\" (UID: \"ce49190c-cda3-4a90-a536-ec1a35e98730\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"
Jul 10 18:59:48.739141 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.739104    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n52wq\" (UniqueName: \"kubernetes.io/projected/ce49190c-cda3-4a90-a536-ec1a35e98730-kube-api-access-n52wq\") pod \"migrator-7f59b54f6c-n5rrx\" (UID: \"ce49190c-cda3-4a90-a536-ec1a35e98730\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"
Jul 10 18:59:48.751299 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.751264    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n52wq\" (UniqueName: \"kubernetes.io/projected/ce49190c-cda3-4a90-a536-ec1a35e98730-kube-api-access-n52wq\") pod \"migrator-7f59b54f6c-n5rrx\" (UID: \"ce49190c-cda3-4a90-a536-ec1a35e98730\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"
Jul 10 18:59:48.874145 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:48.874102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"
Jul 10 18:59:49.087258 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:49.087219    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx"]
Jul 10 18:59:49.093032 ip-10-0-134-26 kubenswrapper[2570]: W0710 18:59:49.092999    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce49190c_cda3_4a90_a536_ec1a35e98730.slice/crio-fd75db19fa3816f147281c2549e747ec3ce38a116dba1711258feab2956c8372 WatchSource:0}: Error finding container fd75db19fa3816f147281c2549e747ec3ce38a116dba1711258feab2956c8372: Status 404 returned error can't find the container with id fd75db19fa3816f147281c2549e747ec3ce38a116dba1711258feab2956c8372
Jul 10 18:59:49.699596 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:49.699561    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx" event={"ID":"ce49190c-cda3-4a90-a536-ec1a35e98730","Type":"ContainerStarted","Data":"fd75db19fa3816f147281c2549e747ec3ce38a116dba1711258feab2956c8372"}
Jul 10 18:59:50.705056 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:50.705017    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8" event={"ID":"71457184-530b-4062-9819-7b5d418b8d37","Type":"ContainerStarted","Data":"154c0f90a29a41f3cf64e80da1f041bc27ce903440af53591e2eea17c519dd97"}
Jul 10 18:59:50.705479 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:50.705211    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:50.706742 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:50.706721    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8"
Jul 10 18:59:50.732331 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:50.732268    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78c78c884f-lhls8" podStartSLOduration=2.59863446 podStartE2EDuration="5.732247996s" totalImagesPullingTime="3.133613536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:50.730875137 +0000 UTC m=+74.864805803" watchObservedRunningTime="2026-07-10 18:59:50.732247996 +0000 UTC m=+74.866178665"
Jul 10 18:59:51.709374 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:51.709331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx" event={"ID":"ce49190c-cda3-4a90-a536-ec1a35e98730","Type":"ContainerStarted","Data":"8c0deb1d8cb8f23fe5b9ec71edbf37202e38645892317219b8ecd6b05edcafe7"}
Jul 10 18:59:51.709882 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:51.709401    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx" event={"ID":"ce49190c-cda3-4a90-a536-ec1a35e98730","Type":"ContainerStarted","Data":"a439f4f6546e74f95b855ae91260ddfdc6747ef9d61190a00616a27ada29211a"}
Jul 10 18:59:51.773468 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:51.773419    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n5rrx" podStartSLOduration=2.030419021 podStartE2EDuration="3.773402639s" totalImagesPullingTime="1.742983618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:51.772959546 +0000 UTC m=+75.906890214" watchObservedRunningTime="2026-07-10 18:59:51.773402639 +0000 UTC m=+75.907333305"
Jul 10 18:59:52.368849 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:52.368806    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:52.368849 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:52.368865    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 18:59:52.369080 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:52.368944    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:59:52.369080 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:52.368962    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:52.369080 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:52.369016    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:00.369002464 +0000 UTC m=+84.502933109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:59:52.369080 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:52.369029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:00.369022258 +0000 UTC m=+84.502952903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : configmap "kube-rbac-proxy" not found
Jul 10 18:59:55.999044 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:55.999010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:55.999044 ip-10-0-134-26 kubenswrapper[2570]: I0710 18:59:55.999062    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 18:59:55.999568 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:55.999159    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 10 18:59:55.999568 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:55.999202    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 19:00:11.999179258 +0000 UTC m=+96.133109903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : configmap references non-existent config key: service-ca.crt
Jul 10 18:59:55.999568 ip-10-0-134-26 kubenswrapper[2570]: E0710 18:59:55.999226    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs podName:846ea7b0-fbdc-4a3c-93d7-be041d80210a nodeName:}" failed. No retries permitted until 2026-07-10 19:00:11.999218913 +0000 UTC m=+96.133149559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs") pod "router-default-648566b76d-5cwnc" (UID: "846ea7b0-fbdc-4a3c-93d7-be041d80210a") : secret "router-metrics-certs-default" not found
Jul 10 19:00:00.436346 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:00.436231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:00.436346 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:00.436284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:00.436715 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:00:00.436423    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 19:00:00.436715 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:00:00.436505    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls podName:bc1193ca-6386-46ad-9f2f-0acc30d02996 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:16.436486087 +0000 UTC m=+100.570416750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-blkm7" (UID: "bc1193ca-6386-46ad-9f2f-0acc30d02996") : secret "insights-runtime-extractor-tls" not found
Jul 10 19:00:00.436888 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:00.436867    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc1193ca-6386-46ad-9f2f-0acc30d02996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:12.021497 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.021458    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:12.021927 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.021514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:12.022112 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.022091    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846ea7b0-fbdc-4a3c-93d7-be041d80210a-service-ca-bundle\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:12.023842 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.023821    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846ea7b0-fbdc-4a3c-93d7-be041d80210a-metrics-certs\") pod \"router-default-648566b76d-5cwnc\" (UID: \"846ea7b0-fbdc-4a3c-93d7-be041d80210a\") " pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:12.287577 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.287463    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:12.421213 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.421192    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648566b76d-5cwnc"]
Jul 10 19:00:12.423615 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:12.423583    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846ea7b0_fbdc_4a3c_93d7_be041d80210a.slice/crio-67217e2b53a27b678030f76a5342c619c3bd26448ab633d5d29f5735f093394b WatchSource:0}: Error finding container 67217e2b53a27b678030f76a5342c619c3bd26448ab633d5d29f5735f093394b: Status 404 returned error can't find the container with id 67217e2b53a27b678030f76a5342c619c3bd26448ab633d5d29f5735f093394b
Jul 10 19:00:12.773622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.773586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648566b76d-5cwnc" event={"ID":"846ea7b0-fbdc-4a3c-93d7-be041d80210a","Type":"ContainerStarted","Data":"0a95c0d251d00fcf54d117f8073244a175c69d4a7eabc79d4b3c195544b42344"}
Jul 10 19:00:12.773817 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:12.773643    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648566b76d-5cwnc" event={"ID":"846ea7b0-fbdc-4a3c-93d7-be041d80210a","Type":"ContainerStarted","Data":"67217e2b53a27b678030f76a5342c619c3bd26448ab633d5d29f5735f093394b"}
Jul 10 19:00:13.288017 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:13.287983    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:13.290585 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:13.290562    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:13.330363 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:13.330318    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-648566b76d-5cwnc" podStartSLOduration=33.330300496 podStartE2EDuration="33.330300496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:12.805375021 +0000 UTC m=+96.939305688" watchObservedRunningTime="2026-07-10 19:00:13.330300496 +0000 UTC m=+97.464231162"
Jul 10 19:00:13.777237 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:13.777207    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:13.778422 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:13.778403    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-648566b76d-5cwnc"
Jul 10 19:00:15.146542 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.146502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 19:00:15.148794 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.148775    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d9bdc1d-3a17-4970-992f-fc348fd4a8e1-metrics-tls\") pod \"dns-default-mllg9\" (UID: \"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1\") " pod="openshift-dns/dns-default-mllg9"
Jul 10 19:00:15.431988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.431905    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4zmd\""
Jul 10 19:00:15.439426 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.439406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mllg9"
Jul 10 19:00:15.571165 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.570991    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mllg9"]
Jul 10 19:00:15.573061 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:15.573028    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9bdc1d_3a17_4970_992f_fc348fd4a8e1.slice/crio-b7ce6393903bb06fce3f6f9ef99e499be57772e4e1da642d3cdf78fd35701dda WatchSource:0}: Error finding container b7ce6393903bb06fce3f6f9ef99e499be57772e4e1da642d3cdf78fd35701dda: Status 404 returned error can't find the container with id b7ce6393903bb06fce3f6f9ef99e499be57772e4e1da642d3cdf78fd35701dda
Jul 10 19:00:15.783631 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:15.783537    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mllg9" event={"ID":"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1","Type":"ContainerStarted","Data":"b7ce6393903bb06fce3f6f9ef99e499be57772e4e1da642d3cdf78fd35701dda"}
Jul 10 19:00:16.234652 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.234612    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"]
Jul 10 19:00:16.241659 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.241624    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:16.254439 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.254413    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nrxcg\""
Jul 10 19:00:16.323623 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.323557    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 10 19:00:16.341444 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.341410    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"]
Jul 10 19:00:16.342626 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.342601    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7fc75cb8-46vdg"]
Jul 10 19:00:16.345818 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.345791    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.356068 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.356042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bf7a3746-41b3-4385-a1e3-3d38ae093ad1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-xmgwj\" (UID: \"bf7a3746-41b3-4385-a1e3-3d38ae093ad1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:16.362528 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.362507    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 19:00:16.382154 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.382121    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 19:00:16.383965 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.383936    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 19:00:16.385465 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.385447    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68v4l\""
Jul 10 19:00:16.456969 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.456931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bf7a3746-41b3-4385-a1e3-3d38ae093ad1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-xmgwj\" (UID: \"bf7a3746-41b3-4385-a1e3-3d38ae093ad1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:16.457206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457005    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-registry-tls\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:16.457206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-trusted-ca\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457076    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cwq\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-kube-api-access-g2cwq\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-bound-sa-token\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457198    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-installation-pull-secrets\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15bd5471-1d37-403d-bea8-db0218335df7-ca-trust-extracted\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-image-registry-private-configuration\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.457423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.457319    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-registry-certificates\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.459743 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.459715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc1193ca-6386-46ad-9f2f-0acc30d02996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blkm7\" (UID: \"bc1193ca-6386-46ad-9f2f-0acc30d02996\") " pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:16.459881 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.459753    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bf7a3746-41b3-4385-a1e3-3d38ae093ad1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-xmgwj\" (UID: \"bf7a3746-41b3-4385-a1e3-3d38ae093ad1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:16.479103 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.479070    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 19:00:16.480382 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.480359    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7fc75cb8-46vdg"]
Jul 10 19:00:16.552406 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.552330    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:16.558572 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558547    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-trusted-ca\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.558713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cwq\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-kube-api-access-g2cwq\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.558713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-bound-sa-token\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.558713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558646    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-installation-pull-secrets\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.558713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15bd5471-1d37-403d-bea8-db0218335df7-ca-trust-extracted\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.558713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558698    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-image-registry-private-configuration\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.559047 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.558720    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-registry-certificates\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.559263 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.559233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-registry-tls\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.559376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.559321    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15bd5471-1d37-403d-bea8-db0218335df7-ca-trust-extracted\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.559606 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.559584    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-trusted-ca\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.560457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.560434    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15bd5471-1d37-403d-bea8-db0218335df7-registry-certificates\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.562272 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.562140    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-image-registry-private-configuration\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.562384 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.562367    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-registry-tls\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.562480 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.562461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15bd5471-1d37-403d-bea8-db0218335df7-installation-pull-secrets\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.615282 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.615210    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-bound-sa-token\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.625643 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.621921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cwq\" (UniqueName: \"kubernetes.io/projected/15bd5471-1d37-403d-bea8-db0218335df7-kube-api-access-g2cwq\") pod \"image-registry-5d7fc75cb8-46vdg\" (UID: \"15bd5471-1d37-403d-bea8-db0218335df7\") " pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.656118 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.656084    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:16.706715 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.706526    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-blkm7"
Jul 10 19:00:16.730925 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.730895    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"]
Jul 10 19:00:16.812157 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:16.812086    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7fc75cb8-46vdg"]
Jul 10 19:00:17.187862 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:17.187831    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7a3746_41b3_4385_a1e3_3d38ae093ad1.slice/crio-b61d48c35141c2d85f65cb621bc4101a2712c12feca39b12d8422c37207bb0f2 WatchSource:0}: Error finding container b61d48c35141c2d85f65cb621bc4101a2712c12feca39b12d8422c37207bb0f2: Status 404 returned error can't find the container with id b61d48c35141c2d85f65cb621bc4101a2712c12feca39b12d8422c37207bb0f2
Jul 10 19:00:17.188447 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:17.188418    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bd5471_1d37_403d_bea8_db0218335df7.slice/crio-1518b32284ecb4e8b16652d41bcf0e68cb569a64a21b69957b75cef0cb9d87fc WatchSource:0}: Error finding container 1518b32284ecb4e8b16652d41bcf0e68cb569a64a21b69957b75cef0cb9d87fc: Status 404 returned error can't find the container with id 1518b32284ecb4e8b16652d41bcf0e68cb569a64a21b69957b75cef0cb9d87fc
Jul 10 19:00:17.317198 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.317164    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-blkm7"]
Jul 10 19:00:17.321029 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:17.320998    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1193ca_6386_46ad_9f2f_0acc30d02996.slice/crio-7ee128e6db3e961e968fcfb6c8bdc5df0d3a306ee44eb730b8a54c0c0c7139b5 WatchSource:0}: Error finding container 7ee128e6db3e961e968fcfb6c8bdc5df0d3a306ee44eb730b8a54c0c0c7139b5: Status 404 returned error can't find the container with id 7ee128e6db3e961e968fcfb6c8bdc5df0d3a306ee44eb730b8a54c0c0c7139b5
Jul 10 19:00:17.793719 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.793631    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blkm7" event={"ID":"bc1193ca-6386-46ad-9f2f-0acc30d02996","Type":"ContainerStarted","Data":"7ee128e6db3e961e968fcfb6c8bdc5df0d3a306ee44eb730b8a54c0c0c7139b5"}
Jul 10 19:00:17.795935 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.795903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mllg9" event={"ID":"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1","Type":"ContainerStarted","Data":"7e0e70054c7b754104e6c282f5f1d6c89697ef0189eac59431860317e8c9df2d"}
Jul 10 19:00:17.796053 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.795971    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mllg9" event={"ID":"4d9bdc1d-3a17-4970-992f-fc348fd4a8e1","Type":"ContainerStarted","Data":"91e9e38f1ab6d4ee6a3b5937eaacf2baa4ae309dbc397bd8e1541da651eb1139"}
Jul 10 19:00:17.796110 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.796076    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mllg9"
Jul 10 19:00:17.798291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.798228    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg" event={"ID":"15bd5471-1d37-403d-bea8-db0218335df7","Type":"ContainerStarted","Data":"f4a40e0177c3c6c9e57e391bc1cbc44602ad25d14174df9d9f74f5dff4cba186"}
Jul 10 19:00:17.798291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.798265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg" event={"ID":"15bd5471-1d37-403d-bea8-db0218335df7","Type":"ContainerStarted","Data":"1518b32284ecb4e8b16652d41bcf0e68cb569a64a21b69957b75cef0cb9d87fc"}
Jul 10 19:00:17.798457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.798392    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:17.799822 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.799800    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj" event={"ID":"bf7a3746-41b3-4385-a1e3-3d38ae093ad1","Type":"ContainerStarted","Data":"b61d48c35141c2d85f65cb621bc4101a2712c12feca39b12d8422c37207bb0f2"}
Jul 10 19:00:17.825063 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.824099    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mllg9" podStartSLOduration=64.895600002 podStartE2EDuration="1m6.824075992s" totalImagesPullingTime="1.92847599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:17.823005021 +0000 UTC m=+101.956935688" watchObservedRunningTime="2026-07-10 19:00:17.824075992 +0000 UTC m=+101.958006659"
Jul 10 19:00:17.858292 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:17.857849    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg" podStartSLOduration=1.857828896 podStartE2EDuration="1.857828896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:17.856445218 +0000 UTC m=+101.990375887" watchObservedRunningTime="2026-07-10 19:00:17.857828896 +0000 UTC m=+101.991759545"
Jul 10 19:00:18.698799 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:18.698732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jg5tw"
Jul 10 19:00:21.817260 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:21.817225    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj" event={"ID":"bf7a3746-41b3-4385-a1e3-3d38ae093ad1","Type":"ContainerStarted","Data":"a9493881ab2c0d8bba026e4182131646509b9644511de435fc2ef23edf6c29c3"}
Jul 10 19:00:21.817645 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:21.817400    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:21.821839 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:21.821816    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj"
Jul 10 19:00:21.837444 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:21.837356    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-xmgwj" podStartSLOduration=1.456299482 podStartE2EDuration="5.837341698s" totalImagesPullingTime="4.381042216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:21.836342164 +0000 UTC m=+105.970272835" watchObservedRunningTime="2026-07-10 19:00:21.837341698 +0000 UTC m=+105.971272365"
Jul 10 19:00:22.252283 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.252251    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hnhtd"]
Jul 10 19:00:22.268155 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.268128    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hnhtd"]
Jul 10 19:00:22.268315 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.268241    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.272048 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.271983    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 10 19:00:22.272048 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.272025    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tpxq2\""
Jul 10 19:00:22.272299 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.272057    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 19:00:22.272299 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.271992    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 19:00:22.272299 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.271992    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 19:00:22.272299 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.271992    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 10 19:00:22.408757 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.408711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.408937 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.408828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6400aba-269b-456a-a1a5-13b37504ecbc-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.408937 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.408905    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.408937 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.408922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkcv\" (UniqueName: \"kubernetes.io/projected/e6400aba-269b-456a-a1a5-13b37504ecbc-kube-api-access-7mkcv\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.510116 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.510017    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.510116 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.510059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkcv\" (UniqueName: \"kubernetes.io/projected/e6400aba-269b-456a-a1a5-13b37504ecbc-kube-api-access-7mkcv\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.510116 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.510092    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.510116 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.510118    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6400aba-269b-456a-a1a5-13b37504ecbc-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.510865 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.510845    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6400aba-269b-456a-a1a5-13b37504ecbc-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.512539 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.512515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.512624 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.512564    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6400aba-269b-456a-a1a5-13b37504ecbc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.519976 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.519952    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkcv\" (UniqueName: \"kubernetes.io/projected/e6400aba-269b-456a-a1a5-13b37504ecbc-kube-api-access-7mkcv\") pod \"prometheus-operator-64785b6675-hnhtd\" (UID: \"e6400aba-269b-456a-a1a5-13b37504ecbc\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.577173 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.577129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd"
Jul 10 19:00:22.718752 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.718647    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hnhtd"]
Jul 10 19:00:22.774419 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:22.774338    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6400aba_269b_456a_a1a5_13b37504ecbc.slice/crio-61bfe2894a7e4f49e595e58857dc3e88f679a3867033d0698cf45979eea2c401 WatchSource:0}: Error finding container 61bfe2894a7e4f49e595e58857dc3e88f679a3867033d0698cf45979eea2c401: Status 404 returned error can't find the container with id 61bfe2894a7e4f49e595e58857dc3e88f679a3867033d0698cf45979eea2c401
Jul 10 19:00:22.821774 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.821731    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blkm7" event={"ID":"bc1193ca-6386-46ad-9f2f-0acc30d02996","Type":"ContainerStarted","Data":"3551d21f3d31c339cd3729c0c651c1813bf8288cc8eb20d2e4718a3dc84977c7"}
Jul 10 19:00:22.822818 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:22.822795    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd" event={"ID":"e6400aba-269b-456a-a1a5-13b37504ecbc","Type":"ContainerStarted","Data":"61bfe2894a7e4f49e595e58857dc3e88f679a3867033d0698cf45979eea2c401"}
Jul 10 19:00:23.829996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:23.829958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blkm7" event={"ID":"bc1193ca-6386-46ad-9f2f-0acc30d02996","Type":"ContainerStarted","Data":"864c3920ed2725ef893f96d1c23a92483e6c03baa4e255ecb427c6a340c4feb7"}
Jul 10 19:00:24.835992 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:24.835957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd" event={"ID":"e6400aba-269b-456a-a1a5-13b37504ecbc","Type":"ContainerStarted","Data":"5bedda292089665696a62a787ab6c252f33c72dc2cd7b373bfcb3e007d059b4c"}
Jul 10 19:00:26.845779 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.845717    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd" event={"ID":"e6400aba-269b-456a-a1a5-13b37504ecbc","Type":"ContainerStarted","Data":"2c02cbba4e1e537c33bce0828f7973dd84d5441d3e92da6dd99cf7d72d4adc99"}
Jul 10 19:00:26.847644 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.847618    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blkm7" event={"ID":"bc1193ca-6386-46ad-9f2f-0acc30d02996","Type":"ContainerStarted","Data":"0d03372ebfd41be5fcf5d568764e153a9c9f09c8c14b11135260b106845ca5b4"}
Jul 10 19:00:26.872113 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.872083    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-v965r"]
Jul 10 19:00:26.872654 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.872608    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-hnhtd" podStartSLOduration=2.9735266940000002 podStartE2EDuration="4.872592513s" totalImagesPullingTime="1.899065819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:26.870240159 +0000 UTC m=+111.004170825" watchObservedRunningTime="2026-07-10 19:00:26.872592513 +0000 UTC m=+111.006523179"
Jul 10 19:00:26.874108 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.874089    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:26.876892 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.876867    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ksvsl\""
Jul 10 19:00:26.877029 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.876869    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 10 19:00:26.877258 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.877235    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 10 19:00:26.897327 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.897285    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-v965r"]
Jul 10 19:00:26.900849 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:26.900794    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-blkm7" podStartSLOduration=34.588255119 podStartE2EDuration="42.900775334s" totalImagesPullingTime="8.312520215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:26.899816367 +0000 UTC m=+111.033747033" watchObservedRunningTime="2026-07-10 19:00:26.900775334 +0000 UTC m=+111.034706001"
Jul 10 19:00:27.051229 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.051184    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjp7q\" (UniqueName: \"kubernetes.io/projected/62f063d8-32d1-4393-9c83-06f9a37f23af-kube-api-access-jjp7q\") pod \"downloads-68c9cbb497-v965r\" (UID: \"62f063d8-32d1-4393-9c83-06f9a37f23af\") " pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:27.152592 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.152554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjp7q\" (UniqueName: \"kubernetes.io/projected/62f063d8-32d1-4393-9c83-06f9a37f23af-kube-api-access-jjp7q\") pod \"downloads-68c9cbb497-v965r\" (UID: \"62f063d8-32d1-4393-9c83-06f9a37f23af\") " pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:27.161589 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.161565    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjp7q\" (UniqueName: \"kubernetes.io/projected/62f063d8-32d1-4393-9c83-06f9a37f23af-kube-api-access-jjp7q\") pod \"downloads-68c9cbb497-v965r\" (UID: \"62f063d8-32d1-4393-9c83-06f9a37f23af\") " pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:27.183435 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.183391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:27.328099 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.328069    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-v965r"]
Jul 10 19:00:27.331474 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:27.331445    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f063d8_32d1_4393_9c83_06f9a37f23af.slice/crio-5f2e18b30c90e26fd463ea49150e116deccfe3a2e723581f59789e792de5c397 WatchSource:0}: Error finding container 5f2e18b30c90e26fd463ea49150e116deccfe3a2e723581f59789e792de5c397: Status 404 returned error can't find the container with id 5f2e18b30c90e26fd463ea49150e116deccfe3a2e723581f59789e792de5c397
Jul 10 19:00:27.805924 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.805895    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mllg9"
Jul 10 19:00:27.852304 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:27.852268    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-v965r" event={"ID":"62f063d8-32d1-4393-9c83-06f9a37f23af","Type":"ContainerStarted","Data":"5f2e18b30c90e26fd463ea49150e116deccfe3a2e723581f59789e792de5c397"}
Jul 10 19:00:28.642541 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.642510    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-98f2r"]
Jul 10 19:00:28.645436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.645376    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.649545 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.649520    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 19:00:28.649882 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.649862    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 19:00:28.650280 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.650258    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 19:00:28.650494 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.650479    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6xvsw\""
Jul 10 19:00:28.767051 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767020    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-root\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767051 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767079    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767181    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-sys\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-wtmp\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-metrics-client-ca\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767290    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-textfile\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9w2\" (UniqueName: \"kubernetes.io/projected/2328632a-950d-48a8-a941-5bc1e01090b8-kube-api-access-mb9w2\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.767473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.767389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-tls\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868508    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-sys\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868541    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-wtmp\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868592    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-metrics-client-ca\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868619    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-textfile\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9w2\" (UniqueName: \"kubernetes.io/projected/2328632a-950d-48a8-a941-5bc1e01090b8-kube-api-access-mb9w2\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868643    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-sys\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868671    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-tls\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868709    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-root\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.868946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.868725    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-wtmp\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.869374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.869043    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2328632a-950d-48a8-a941-5bc1e01090b8-root\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.869374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.869155    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.869374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.869171    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2328632a-950d-48a8-a941-5bc1e01090b8-metrics-client-ca\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.869374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.869226    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-textfile\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.871403 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.871369    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-tls\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.871585 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.871567    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2328632a-950d-48a8-a941-5bc1e01090b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.877554 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.877528    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9w2\" (UniqueName: \"kubernetes.io/projected/2328632a-950d-48a8-a941-5bc1e01090b8-kube-api-access-mb9w2\") pod \"node-exporter-98f2r\" (UID: \"2328632a-950d-48a8-a941-5bc1e01090b8\") " pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.956633 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:28.956543    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-98f2r"
Jul 10 19:00:28.966944 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:28.966909    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2328632a_950d_48a8_a941_5bc1e01090b8.slice/crio-058db25c0707921e9c00f8b2f415c3b441e998816e2aff79222bd7e7c265abdb WatchSource:0}: Error finding container 058db25c0707921e9c00f8b2f415c3b441e998816e2aff79222bd7e7c265abdb: Status 404 returned error can't find the container with id 058db25c0707921e9c00f8b2f415c3b441e998816e2aff79222bd7e7c265abdb
Jul 10 19:00:29.863408 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:29.863349    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-98f2r" event={"ID":"2328632a-950d-48a8-a941-5bc1e01090b8","Type":"ContainerStarted","Data":"058db25c0707921e9c00f8b2f415c3b441e998816e2aff79222bd7e7c265abdb"}
Jul 10 19:00:30.870155 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:30.870115    2570 generic.go:356] "Generic (PLEG): container finished" podID="2328632a-950d-48a8-a941-5bc1e01090b8" containerID="6a884a4d3872dc3c54fc6ff4bc41b194f236a3b31066d186656692005523cb37" exitCode=0
Jul 10 19:00:30.871003 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:30.870177    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-98f2r" event={"ID":"2328632a-950d-48a8-a941-5bc1e01090b8","Type":"ContainerDied","Data":"6a884a4d3872dc3c54fc6ff4bc41b194f236a3b31066d186656692005523cb37"}
Jul 10 19:00:31.761414 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.761372    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"]
Jul 10 19:00:31.764420 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.764395    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.767716 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.767687    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 10 19:00:31.767866 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.767750    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 10 19:00:31.767866 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.767822    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 10 19:00:31.767866 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.767833    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7gqjarfj5912n\""
Jul 10 19:00:31.768106 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.768084    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 10 19:00:31.768211 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.768134    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2dtmd\""
Jul 10 19:00:31.769376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.769359    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 10 19:00:31.777436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.777395    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"]
Jul 10 19:00:31.795309 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.795274    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.795501 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.795426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.795501 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.795466    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.796105 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.796074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.796222 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.796158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.796222 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.796203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-grpc-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.796345 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.796229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqjc\" (UniqueName: \"kubernetes.io/projected/4605d126-eb55-4e5c-83e7-1f94eaf58f04-kube-api-access-hbqjc\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.796345 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.796275    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4605d126-eb55-4e5c-83e7-1f94eaf58f04-metrics-client-ca\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.876436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.876328    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-98f2r" event={"ID":"2328632a-950d-48a8-a941-5bc1e01090b8","Type":"ContainerStarted","Data":"0b66a1b4ae2ab1d2c5fa68cf66052b718bb92d2232a5ee0c961d8a3b9d7d71bb"}
Jul 10 19:00:31.876436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.876402    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-98f2r" event={"ID":"2328632a-950d-48a8-a941-5bc1e01090b8","Type":"ContainerStarted","Data":"126aa6fb72463cf722b550c0a41a73d57235a00dc75dad9cac8a6b76c7a27e72"}
Jul 10 19:00:31.897300 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897352    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897485    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897524    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-grpc-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897738 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897597    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqjc\" (UniqueName: \"kubernetes.io/projected/4605d126-eb55-4e5c-83e7-1f94eaf58f04-kube-api-access-hbqjc\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.897738 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.897645    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4605d126-eb55-4e5c-83e7-1f94eaf58f04-metrics-client-ca\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.898567 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.898434    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4605d126-eb55-4e5c-83e7-1f94eaf58f04-metrics-client-ca\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.901069 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.900702    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.901605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.901202    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.901605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.901325    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.901605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.901564    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.902172 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.902148    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-grpc-tls\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.903296 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.902885    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-98f2r" podStartSLOduration=2.806959085 podStartE2EDuration="3.902867285s" totalImagesPullingTime="1.019604949s" totalInitContainerRuntime="76.303251ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:31.901370531 +0000 UTC m=+116.035301199" watchObservedRunningTime="2026-07-10 19:00:31.902867285 +0000 UTC m=+116.036797953"
Jul 10 19:00:31.903296 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.902940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4605d126-eb55-4e5c-83e7-1f94eaf58f04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:31.908036 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:31.907994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqjc\" (UniqueName: \"kubernetes.io/projected/4605d126-eb55-4e5c-83e7-1f94eaf58f04-kube-api-access-hbqjc\") pod \"thanos-querier-674d8c9cbb-m9xgd\" (UID: \"4605d126-eb55-4e5c-83e7-1f94eaf58f04\") " pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:32.075866 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:32.075775    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:32.238058 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:32.238023    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"]
Jul 10 19:00:32.241416 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:32.241384    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4605d126_eb55_4e5c_83e7_1f94eaf58f04.slice/crio-873b947053609f019b6c203c9205ae9a247780f6acd1eaaa1abb731e6c955227 WatchSource:0}: Error finding container 873b947053609f019b6c203c9205ae9a247780f6acd1eaaa1abb731e6c955227: Status 404 returned error can't find the container with id 873b947053609f019b6c203c9205ae9a247780f6acd1eaaa1abb731e6c955227
Jul 10 19:00:32.883843 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:32.883720    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"873b947053609f019b6c203c9205ae9a247780f6acd1eaaa1abb731e6c955227"}
Jul 10 19:00:34.901679 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:34.901511    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"c98728af9752ea07573033270eaa6d2f1b1786e51299a5a1c74f6cbdca425767"}
Jul 10 19:00:34.901679 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:34.901606    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"e0b94b479d17817516cae50566af793e3a4434dcda3afe6a763ed120cfda3658"}
Jul 10 19:00:34.901679 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:34.901642    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"21e2165389195247f5f57cc5b801df00df13342335f1acd8c48cc3952732953f"}
Jul 10 19:00:35.909095 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:35.908978    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"f373df22550c152e6e9e33544afb4d71eab5e9645ca3438c6f2be582185d7dfa"}
Jul 10 19:00:35.909095 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:35.909064    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"d5787b52f28e456eeae6b4d07402d79e6643153fe6587f684269f3229b68806a"}
Jul 10 19:00:35.909095 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:35.909102    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" event={"ID":"4605d126-eb55-4e5c-83e7-1f94eaf58f04","Type":"ContainerStarted","Data":"f67999424e583267b61f64ce6ab365b1c6da2cc9d0e91386ad5ae9d5f1f44590"}
Jul 10 19:00:35.909655 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:35.909139    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:35.953167 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:35.953102    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd" podStartSLOduration=2.187755802 podStartE2EDuration="4.953078784s" totalImagesPullingTime="2.765322982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:35.950549891 +0000 UTC m=+120.084480560" watchObservedRunningTime="2026-07-10 19:00:35.953078784 +0000 UTC m=+120.087009451"
Jul 10 19:00:38.808610 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:38.808577    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7fc75cb8-46vdg"
Jul 10 19:00:41.923390 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:41.923357    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-674d8c9cbb-m9xgd"
Jul 10 19:00:43.946154 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:43.946116    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-v965r" event={"ID":"62f063d8-32d1-4393-9c83-06f9a37f23af","Type":"ContainerStarted","Data":"b1ccd7f39fb13a79afec3f7c6f1fdf8fecb33b821631d79c245411a965b391a8"}
Jul 10 19:00:43.946629 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:43.946337    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:43.958118 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:43.958090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-v965r"
Jul 10 19:00:43.966533 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:43.966482    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-v965r" podStartSLOduration=1.697018777 podStartE2EDuration="17.966463314s" totalImagesPullingTime="16.269444537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:43.964370948 +0000 UTC m=+128.098301618" watchObservedRunningTime="2026-07-10 19:00:43.966463314 +0000 UTC m=+128.100393981"
Jul 10 19:00:46.240573 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.240532    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 19:00:46.243343 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.243309    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f255b1-7837-4251-9bca-3aa6563d4333-metrics-certs\") pod \"network-metrics-daemon-8qdk8\" (UID: \"a0f255b1-7837-4251-9bca-3aa6563d4333\") " pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 19:00:46.533391 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.533304    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvrkc\""
Jul 10 19:00:46.541029 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.540992    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qdk8"
Jul 10 19:00:46.731609 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.731386    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8qdk8"]
Jul 10 19:00:46.735011 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:00:46.734970    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f255b1_7837_4251_9bca_3aa6563d4333.slice/crio-1219ab898f0744078825a81b0a8a946d2a77daee81708029b797bdea9b1dbbca WatchSource:0}: Error finding container 1219ab898f0744078825a81b0a8a946d2a77daee81708029b797bdea9b1dbbca: Status 404 returned error can't find the container with id 1219ab898f0744078825a81b0a8a946d2a77daee81708029b797bdea9b1dbbca
Jul 10 19:00:46.959624 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:46.959581    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qdk8" event={"ID":"a0f255b1-7837-4251-9bca-3aa6563d4333","Type":"ContainerStarted","Data":"1219ab898f0744078825a81b0a8a946d2a77daee81708029b797bdea9b1dbbca"}
Jul 10 19:00:48.971676 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:48.971636    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qdk8" event={"ID":"a0f255b1-7837-4251-9bca-3aa6563d4333","Type":"ContainerStarted","Data":"860edd4b99c91bf541596117d933ca6c2ca397bc94cdfdc3afa0a43c47738178"}
Jul 10 19:00:48.972202 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:48.971702    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qdk8" event={"ID":"a0f255b1-7837-4251-9bca-3aa6563d4333","Type":"ContainerStarted","Data":"2794818f7dbb05489bb3b154b5873d91c15401645591cb35f61ba9830eca1d0b"}
Jul 10 19:00:49.003064 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:00:49.003008    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8qdk8" podStartSLOduration=130.469666865 podStartE2EDuration="2m12.002990178s" totalImagesPullingTime="1.533323313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:49.002286671 +0000 UTC m=+133.136217340" watchObservedRunningTime="2026-07-10 19:00:49.002990178 +0000 UTC m=+133.136920844"
Jul 10 19:01:03.024887 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:01:03.024852    2570 generic.go:356] "Generic (PLEG): container finished" podID="256d0b26-9b34-46d7-9b6e-e05911b13b66" containerID="2eb95a542bf05f02397253b6ef41174e61711b3a6b949370219a574904433205" exitCode=0
Jul 10 19:01:03.025396 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:01:03.024929    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6" event={"ID":"256d0b26-9b34-46d7-9b6e-e05911b13b66","Type":"ContainerDied","Data":"2eb95a542bf05f02397253b6ef41174e61711b3a6b949370219a574904433205"}
Jul 10 19:01:03.025473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:01:03.025402    2570 scope.go:117] "RemoveContainer" containerID="2eb95a542bf05f02397253b6ef41174e61711b3a6b949370219a574904433205"
Jul 10 19:01:04.031455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:01:04.031425    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-jhgc6" event={"ID":"256d0b26-9b34-46d7-9b6e-e05911b13b66","Type":"ContainerStarted","Data":"bf677aacdde7bb8a9c49f9998a9c2723560ecb510e55377a2a1d09af1becb9dc"}
Jul 10 19:03:36.283042 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:03:36.283005    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:03:36.283622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:03:36.283603    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:03:36.291676 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:03:36.291650    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 19:05:25.351097 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.351066    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-rzrth"]
Jul 10 19:05:25.353393 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.353371    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.358141 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.358113    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nmv7m\""
Jul 10 19:05:25.358141 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.358134    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 10 19:05:25.358325 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.358201    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 10 19:05:25.358325 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.358207    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 10 19:05:25.358839 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.358819    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 10 19:05:25.410912 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.410883    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-rzrth"]
Jul 10 19:05:25.428087 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.428044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb9c\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-kube-api-access-knb9c\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.428416 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.428394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-certificates\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.529568 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.529528    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb9c\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-kube-api-access-knb9c\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.529799 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.529618    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-certificates\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.532298 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.532276    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-certificates\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.550145 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.550112    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb9c\" (UniqueName: \"kubernetes.io/projected/db594b03-7af2-4985-b407-8a097e011412-kube-api-access-knb9c\") pod \"keda-admission-5fbd5c4644-rzrth\" (UID: \"db594b03-7af2-4985-b407-8a097e011412\") " pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.662834 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.662801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:25.842143 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.842112    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-rzrth"]
Jul 10 19:05:25.844209 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:05:25.844180    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb594b03_7af2_4985_b407_8a097e011412.slice/crio-c37a1b8fefbdf63a759012642ac617c9c792c6276226aa16b595e80671f9fcd4 WatchSource:0}: Error finding container c37a1b8fefbdf63a759012642ac617c9c792c6276226aa16b595e80671f9fcd4: Status 404 returned error can't find the container with id c37a1b8fefbdf63a759012642ac617c9c792c6276226aa16b595e80671f9fcd4
Jul 10 19:05:25.845542 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.845521    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:05:25.975713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:25.975619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth" event={"ID":"db594b03-7af2-4985-b407-8a097e011412","Type":"ContainerStarted","Data":"c37a1b8fefbdf63a759012642ac617c9c792c6276226aa16b595e80671f9fcd4"}
Jul 10 19:05:28.989187 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:28.989151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth" event={"ID":"db594b03-7af2-4985-b407-8a097e011412","Type":"ContainerStarted","Data":"4793021b6518fc43bf1dbba9a826b11e9bf7c9bd36ec1a6fedc10f348dbaa065"}
Jul 10 19:05:28.989552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:28.989231    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:49.995486 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:49.995455    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth"
Jul 10 19:05:50.014304 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:05:50.014252    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-rzrth" podStartSLOduration=22.723293508 podStartE2EDuration="25.014233812s" totalImagesPullingTime="2.290940304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:29.05033211 +0000 UTC m=+413.184262776" watchObservedRunningTime="2026-07-10 19:05:50.014233812 +0000 UTC m=+434.148164489"
Jul 10 19:06:32.216886 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.216850    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nqljj"]
Jul 10 19:06:32.220015 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.219991    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.222874 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.222850    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 19:06:32.223999 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.223980    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6pbqs\""
Jul 10 19:06:32.224066 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.224005    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 19:06:32.224127 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.224006    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:06:32.243557 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.243521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdsv\" (UniqueName: \"kubernetes.io/projected/817feb75-ad27-4329-b412-cf087beb1c99-kube-api-access-7mdsv\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.243712 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.243578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/817feb75-ad27-4329-b412-cf087beb1c99-data\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.245939 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.245910    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nqljj"]
Jul 10 19:06:32.344655 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.344606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdsv\" (UniqueName: \"kubernetes.io/projected/817feb75-ad27-4329-b412-cf087beb1c99-kube-api-access-7mdsv\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.344655 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.344667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/817feb75-ad27-4329-b412-cf087beb1c99-data\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.345022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.345006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/817feb75-ad27-4329-b412-cf087beb1c99-data\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.357870 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.357843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdsv\" (UniqueName: \"kubernetes.io/projected/817feb75-ad27-4329-b412-cf087beb1c99-kube-api-access-7mdsv\") pod \"seaweedfs-64568bcd49-nqljj\" (UID: \"817feb75-ad27-4329-b412-cf087beb1c99\") " pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.528574 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.528486    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:32.654380 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:32.654322    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nqljj"]
Jul 10 19:06:32.657111 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:06:32.657070    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817feb75_ad27_4329_b412_cf087beb1c99.slice/crio-f6c914a6664c4cb3fc69b5be7d2514d44660980a6274cfc9dbb751e7ce6bccc1 WatchSource:0}: Error finding container f6c914a6664c4cb3fc69b5be7d2514d44660980a6274cfc9dbb751e7ce6bccc1: Status 404 returned error can't find the container with id f6c914a6664c4cb3fc69b5be7d2514d44660980a6274cfc9dbb751e7ce6bccc1
Jul 10 19:06:33.223712 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:33.223675    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nqljj" event={"ID":"817feb75-ad27-4329-b412-cf087beb1c99","Type":"ContainerStarted","Data":"f6c914a6664c4cb3fc69b5be7d2514d44660980a6274cfc9dbb751e7ce6bccc1"}
Jul 10 19:06:36.582010 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:36.581986    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 19:06:37.239605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:37.239572    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nqljj" event={"ID":"817feb75-ad27-4329-b412-cf087beb1c99","Type":"ContainerStarted","Data":"ff90479b8f4f874711eb5df96ce0fd031f977be0f08e9cf26c1bf227c499bcfc"}
Jul 10 19:06:37.239798 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:37.239694    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:06:37.259394 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:37.259338    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nqljj" podStartSLOduration=1.339415957 podStartE2EDuration="5.25932079s" totalImagesPullingTime="3.919904833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:37.257036946 +0000 UTC m=+481.390967613" watchObservedRunningTime="2026-07-10 19:06:37.25932079 +0000 UTC m=+481.393251456"
Jul 10 19:06:43.245611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:06:43.245582    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nqljj"
Jul 10 19:07:44.101612 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.101570    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-bl9td"]
Jul 10 19:07:44.104885 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.104860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.107848 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.107809    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 10 19:07:44.107948 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.107867    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-hdjq4\""
Jul 10 19:07:44.119399 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.119376    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-g9z8q"]
Jul 10 19:07:44.122610 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.122590    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-bl9td"]
Jul 10 19:07:44.122705 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.122692    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.126027 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.126008    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 10 19:07:44.126731 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.126713    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rs2r4\""
Jul 10 19:07:44.146748 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.146720    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-g9z8q"]
Jul 10 19:07:44.205088 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.205057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5djs\" (UniqueName: \"kubernetes.io/projected/0067a7a1-b862-4014-9265-747567d95aef-kube-api-access-m5djs\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.205266 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.205097    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.205266 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.205123    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfx6\" (UniqueName: \"kubernetes.io/projected/0564b586-91e6-46b8-8644-6e3fc61ce939-kube-api-access-qjfx6\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.205266 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.205193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.306318 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.306277    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.306318 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.306322    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5djs\" (UniqueName: \"kubernetes.io/projected/0067a7a1-b862-4014-9265-747567d95aef-kube-api-access-m5djs\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.306510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.306359    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.306510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.306387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfx6\" (UniqueName: \"kubernetes.io/projected/0564b586-91e6-46b8-8644-6e3fc61ce939-kube-api-access-qjfx6\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.306510 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:07:44.306415    2570 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 10 19:07:44.306510 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:07:44.306490    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert podName:0067a7a1-b862-4014-9265-747567d95aef nodeName:}" failed. No retries permitted until 2026-07-10 19:07:44.806472323 +0000 UTC m=+548.940402986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert") pod "odh-model-controller-84b54b7f97-g9z8q" (UID: "0067a7a1-b862-4014-9265-747567d95aef") : secret "odh-model-controller-webhook-cert" not found
Jul 10 19:07:44.306510 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:07:44.306497    2570 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 10 19:07:44.306685 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:07:44.306557    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs podName:0564b586-91e6-46b8-8644-6e3fc61ce939 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:44.806543908 +0000 UTC m=+548.940474554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs") pod "model-serving-api-fd65d7d6-bl9td" (UID: "0564b586-91e6-46b8-8644-6e3fc61ce939") : secret "model-serving-api-tls" not found
Jul 10 19:07:44.315232 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.315202    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfx6\" (UniqueName: \"kubernetes.io/projected/0564b586-91e6-46b8-8644-6e3fc61ce939-kube-api-access-qjfx6\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.315342 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.315256    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5djs\" (UniqueName: \"kubernetes.io/projected/0067a7a1-b862-4014-9265-747567d95aef-kube-api-access-m5djs\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.810441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.810400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.810630 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.810467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:44.812873 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.812848    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0067a7a1-b862-4014-9265-747567d95aef-cert\") pod \"odh-model-controller-84b54b7f97-g9z8q\" (UID: \"0067a7a1-b862-4014-9265-747567d95aef\") " pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:44.812977 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:44.812852    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564b586-91e6-46b8-8644-6e3fc61ce939-tls-certs\") pod \"model-serving-api-fd65d7d6-bl9td\" (UID: \"0564b586-91e6-46b8-8644-6e3fc61ce939\") " pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:45.014500 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.014465    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:45.032181 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.032157    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:45.149398 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.149375    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-bl9td"]
Jul 10 19:07:45.151438 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:07:45.151405    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0564b586_91e6_46b8_8644_6e3fc61ce939.slice/crio-506e857fd718da7aaa77409100fe24644ee576d997579547079a42dd005e1901 WatchSource:0}: Error finding container 506e857fd718da7aaa77409100fe24644ee576d997579547079a42dd005e1901: Status 404 returned error can't find the container with id 506e857fd718da7aaa77409100fe24644ee576d997579547079a42dd005e1901
Jul 10 19:07:45.164591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.164570    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-g9z8q"]
Jul 10 19:07:45.166568 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:07:45.166545    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0067a7a1_b862_4014_9265_747567d95aef.slice/crio-0a7b0b88552172d9100a6b7dc25e5591db19cfb48a9a6e969149b29c73f9d2a5 WatchSource:0}: Error finding container 0a7b0b88552172d9100a6b7dc25e5591db19cfb48a9a6e969149b29c73f9d2a5: Status 404 returned error can't find the container with id 0a7b0b88552172d9100a6b7dc25e5591db19cfb48a9a6e969149b29c73f9d2a5
Jul 10 19:07:45.480392 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.480356    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-g9z8q" event={"ID":"0067a7a1-b862-4014-9265-747567d95aef","Type":"ContainerStarted","Data":"0a7b0b88552172d9100a6b7dc25e5591db19cfb48a9a6e969149b29c73f9d2a5"}
Jul 10 19:07:45.481397 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:45.481369    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-bl9td" event={"ID":"0564b586-91e6-46b8-8644-6e3fc61ce939","Type":"ContainerStarted","Data":"506e857fd718da7aaa77409100fe24644ee576d997579547079a42dd005e1901"}
Jul 10 19:07:49.500081 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.500021    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-bl9td" event={"ID":"0564b586-91e6-46b8-8644-6e3fc61ce939","Type":"ContainerStarted","Data":"9aa28785dc78d1b5783fa5d1c5e8cd24721abcfa58d14c603183a388eb55def1"}
Jul 10 19:07:49.500567 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.500098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:07:49.501361 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.501340    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-g9z8q" event={"ID":"0067a7a1-b862-4014-9265-747567d95aef","Type":"ContainerStarted","Data":"12425be99187068965076db50bc40754b249fa4f3881b80e37052fc355805c88"}
Jul 10 19:07:49.501473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.501462    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:07:49.517353 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.517310    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-bl9td" podStartSLOduration=1.7864040079999999 podStartE2EDuration="5.517297681s" totalImagesPullingTime="3.730893673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:49.516393816 +0000 UTC m=+553.650324487" watchObservedRunningTime="2026-07-10 19:07:49.517297681 +0000 UTC m=+553.651228347"
Jul 10 19:07:49.533337 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:07:49.533294    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-g9z8q" podStartSLOduration=1.811448715 podStartE2EDuration="5.533280728s" totalImagesPullingTime="3.721832013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:49.532649207 +0000 UTC m=+553.666579883" watchObservedRunningTime="2026-07-10 19:07:49.533280728 +0000 UTC m=+553.667211432"
Jul 10 19:08:00.510498 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:00.510464    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-g9z8q"
Jul 10 19:08:00.512000 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:00.511982    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-bl9td"
Jul 10 19:08:01.314226 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.314190    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mvjtr"]
Jul 10 19:08:01.317912 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.317894    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.325237 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.325217    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mvjtr"]
Jul 10 19:08:01.351717 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.351692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twb5t\" (UniqueName: \"kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.351869 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.351721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.452124 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.452091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twb5t\" (UniqueName: \"kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.452124 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.452139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.452471 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.452452    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.460400 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.460375    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twb5t\" (UniqueName: \"kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t\") pod \"s3-init-mvjtr\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") " pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.636193 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.636106    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mvjtr"
Jul 10 19:08:01.760296 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:01.760272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mvjtr"]
Jul 10 19:08:01.762701 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:08:01.762672    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca0a5de_8047_41f9_b6f6_7e84f9e244f0.slice/crio-52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b WatchSource:0}: Error finding container 52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b: Status 404 returned error can't find the container with id 52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b
Jul 10 19:08:02.553409 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:02.553375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerStarted","Data":"52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b"}
Jul 10 19:08:05.567053 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:05.566964    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerStarted","Data":"4c456dc3d2856fc16061c49cbcee06f87f3441cb5210da21cbc9f4257236b940"}
Jul 10 19:08:07.582903 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:07.582807    2570 generic.go:356] "Generic (PLEG): container finished" podID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerID="4c456dc3d2856fc16061c49cbcee06f87f3441cb5210da21cbc9f4257236b940" exitCode=0
Jul 10 19:08:07.582903 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:07.582856    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerDied","Data":"4c456dc3d2856fc16061c49cbcee06f87f3441cb5210da21cbc9f4257236b940"}
Jul 10 19:08:12.607703 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:12.607669    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerStarted","Data":"08d339c3b7fef0e5df5da3244df1b60c8b7d9b2be5b11255aff960a2b42106c3"}
Jul 10 19:08:12.626850 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:12.626801    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mvjtr" podStartSLOduration=1.651741726 podStartE2EDuration="11.626780918s" totalImagesPullingTime="8.054294054s" totalInitContainerRuntime="1.920745138s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:12.624553269 +0000 UTC m=+576.758483936" watchObservedRunningTime="2026-07-10 19:08:12.626780918 +0000 UTC m=+576.760711578"
Jul 10 19:08:36.309215 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:36.309185    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:08:36.309707 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:08:36.309359    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:09:16.845685 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:16.845648    2570 generic.go:356] "Generic (PLEG): container finished" podID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerID="08d339c3b7fef0e5df5da3244df1b60c8b7d9b2be5b11255aff960a2b42106c3" exitCode=0
Jul 10 19:09:16.846102 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:16.845707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerDied","Data":"08d339c3b7fef0e5df5da3244df1b60c8b7d9b2be5b11255aff960a2b42106c3"}
Jul 10 19:09:17.972890 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:17.972867    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mvjtr"
Jul 10 19:09:18.059261 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.059228    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twb5t\" (UniqueName: \"kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t\") pod \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") "
Jul 10 19:09:18.059451 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.059311    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache\") pod \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\" (UID: \"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0\") "
Jul 10 19:09:18.061474 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.061444    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t" (OuterVolumeSpecName: "kube-api-access-twb5t") pod "9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" (UID: "9ca0a5de-8047-41f9-b6f6-7e84f9e244f0"). InnerVolumeSpecName "kube-api-access-twb5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:09:18.078194 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.078163    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" (UID: "9ca0a5de-8047-41f9-b6f6-7e84f9e244f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:09:18.160685 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.160648    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twb5t\" (UniqueName: \"kubernetes.io/projected/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-kube-api-access-twb5t\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:09:18.160842 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.160693    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca0a5de-8047-41f9-b6f6-7e84f9e244f0-model-cache\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:09:18.854596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.854513    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mvjtr" event={"ID":"9ca0a5de-8047-41f9-b6f6-7e84f9e244f0","Type":"ContainerDied","Data":"52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b"}
Jul 10 19:09:18.854596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.854553    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b08187be4ab6399ad5c84f4f035cc67a01fc761c456937377bb32a874e3d1b"
Jul 10 19:09:18.854596 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:18.854526    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mvjtr"
Jul 10 19:09:30.473083 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473049    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:09:30.473470 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473456    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerName="download-hf-model"
Jul 10 19:09:30.473510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473472    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerName="download-hf-model"
Jul 10 19:09:30.473510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473483    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerName="s3-init"
Jul 10 19:09:30.473510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473491    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerName="s3-init"
Jul 10 19:09:30.473599 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.473576    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca0a5de-8047-41f9-b6f6-7e84f9e244f0" containerName="s3-init"
Jul 10 19:09:30.477121 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.477103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.479869 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.479850    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cc766-predictor-serving-cert\""
Jul 10 19:09:30.479995 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.479975    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wct6r\""
Jul 10 19:09:30.480064 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.479975    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cc766-kube-rbac-proxy-sar-config\""
Jul 10 19:09:30.480117 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.480064    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:09:30.480331 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.480313    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:09:30.480331 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.480319    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:09:30.480432 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.480339    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:09:30.489291 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.489272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:09:30.565548 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.565513    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.565548 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.565548    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.565759 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.565590    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftzx\" (UniqueName: \"kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.565759 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.565622    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.565759 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.565665    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.666973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.666936    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.666973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.666973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667211 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:09:30.667074    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-cc766-predictor-serving-cert" not found
Jul 10 19:09:30.667211 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667100    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftzx\" (UniqueName: \"kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667211 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:09:30.667130    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls podName:79188647-e6db-4318-8c6a-405ae2a43101 nodeName:}" failed. No retries permitted until 2026-07-10 19:09:31.167114515 +0000 UTC m=+655.301045165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls") pod "isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" (UID: "79188647-e6db-4318-8c6a-405ae2a43101") : secret "isvc-raw-sklearn-batcher-cc766-predictor-serving-cert" not found
Jul 10 19:09:30.667211 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667211 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667194    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667559 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667541    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667601    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.667777 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.667747    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:30.675772 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:30.675744    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftzx\" (UniqueName: \"kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:31.171611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.171576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:31.173941 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.173923    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:31.386903 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.386861    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:31.523963 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.523930    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:09:31.527241 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:09:31.527216    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79188647_e6db_4318_8c6a_405ae2a43101.slice/crio-07efb80070efcc5836105b45f8760a4a272b4ef8a7151fc07073704ed4ae617d WatchSource:0}: Error finding container 07efb80070efcc5836105b45f8760a4a272b4ef8a7151fc07073704ed4ae617d: Status 404 returned error can't find the container with id 07efb80070efcc5836105b45f8760a4a272b4ef8a7151fc07073704ed4ae617d
Jul 10 19:09:31.905301 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.905264    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerStarted","Data":"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"}
Jul 10 19:09:31.905498 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:31.905328    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerStarted","Data":"07efb80070efcc5836105b45f8760a4a272b4ef8a7151fc07073704ed4ae617d"}
Jul 10 19:09:35.921246 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:35.921210    2570 generic.go:356] "Generic (PLEG): container finished" podID="79188647-e6db-4318-8c6a-405ae2a43101" containerID="28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a" exitCode=0
Jul 10 19:09:35.921619 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:35.921282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerDied","Data":"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"}
Jul 10 19:09:49.990476 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:49.990444    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerStarted","Data":"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"}
Jul 10 19:09:52.000358 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:52.000322    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerStarted","Data":"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"}
Jul 10 19:09:56.019649 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.019606    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerStarted","Data":"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"}
Jul 10 19:09:56.020138 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.019884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:56.020138 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.019944    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:56.020138 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.019978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:56.021417 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.021378    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:09:56.022097 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.022070    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:09:56.024808 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.024792    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:09:56.067710 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:56.067661    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podStartSLOduration=3.26054631 podStartE2EDuration="26.067647796s" totalImagesPullingTime="19.17885204s" totalInitContainerRuntime="3.628249446s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:56.040667962 +0000 UTC m=+680.174598627" watchObservedRunningTime="2026-07-10 19:09:56.067647796 +0000 UTC m=+680.201578463"
Jul 10 19:09:57.024085 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:57.024042    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:09:57.024507 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:57.024386    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:09:58.027570 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:58.027533    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:09:58.028006 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:09:58.027793    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:08.027943 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:08.027888    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:08.028419 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:08.028335    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:18.027829 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:18.027750    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:18.028297 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:18.028272    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:28.027611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:28.027504    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:28.028060 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:28.027928    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:38.028206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:38.028159    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:38.028591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:38.028561    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:48.027792 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:48.027718    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:48.028267 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:48.028241    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:10:58.027586 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:58.027533    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:10:58.028035 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:10:58.027989    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:08.028656 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:08.028623    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:08.029042 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:08.028851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:15.491557 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.491522    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:11:15.492016 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.491806    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" containerID="cri-o://b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50" gracePeriod=30
Jul 10 19:11:15.492016 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.491904    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" containerID="cri-o://6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599" gracePeriod=30
Jul 10 19:11:15.492122 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.492030    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" containerID="cri-o://30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb" gracePeriod=30
Jul 10 19:11:15.717825 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.717786    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:11:15.720347 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.720320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.723302 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.723274    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3394-kube-rbac-proxy-sar-config\""
Jul 10 19:11:15.723450 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.723402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3394-predictor-serving-cert\""
Jul 10 19:11:15.733057 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.733030    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:11:15.798328 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.798251    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:11:15.800862 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.800843    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.803576 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.803553    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3394-predictor-serving-cert\""
Jul 10 19:11:15.803708 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.803582    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3394-kube-rbac-proxy-sar-config\""
Jul 10 19:11:15.811312 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.811282    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:11:15.829245 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2hg\" (UniqueName: \"kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.829376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.829376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.829444 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.829444 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829418    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.829509 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.829509 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829470    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.829583 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.829617 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.829651 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.829616    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcqm\" (UniqueName: \"kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.930682 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930638    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.930682 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930689    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930738    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930820    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcqm\" (UniqueName: \"kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:15.930917    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-serving-cert: secret "isvc-xgboost-graph-raw-e3394-predictor-serving-cert" not found
Jul 10 19:11:15.930962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.930956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2hg\" (UniqueName: \"kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.931307 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:15.930994    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls podName:c773b2eb-474a-4cd6-98e2-5125175eb3f1 nodeName:}" failed. No retries permitted until 2026-07-10 19:11:16.430971058 +0000 UTC m=+760.564901729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls") pod "isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1") : secret "isvc-xgboost-graph-raw-e3394-predictor-serving-cert" not found
Jul 10 19:11:15.931307 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.931307 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931078    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.931576 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.931657 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.931657 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931587    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.931746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931679    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.931746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.931855 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.931822    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.933293 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.933273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:15.940287 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.940260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcqm\" (UniqueName: \"kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:15.940675 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:15.940659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2hg\" (UniqueName: \"kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg\") pod \"isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:16.020477 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.020429    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:16.030428 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.030405    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:16.157608 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.157586    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:11:16.160287 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:11:16.160254    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168d15a7_e849_4968_bb60_e19259c523a3.slice/crio-d7cac34f53a048aedb560d4971b44705405e75fe599d3946baf503c035c2679b WatchSource:0}: Error finding container d7cac34f53a048aedb560d4971b44705405e75fe599d3946baf503c035c2679b: Status 404 returned error can't find the container with id d7cac34f53a048aedb560d4971b44705405e75fe599d3946baf503c035c2679b
Jul 10 19:11:16.337316 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.337221    2570 generic.go:356] "Generic (PLEG): container finished" podID="79188647-e6db-4318-8c6a-405ae2a43101" containerID="30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb" exitCode=2
Jul 10 19:11:16.337316 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.337295    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerDied","Data":"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"}
Jul 10 19:11:16.338924 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.338891    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerStarted","Data":"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"}
Jul 10 19:11:16.339054 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.338961    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerStarted","Data":"d7cac34f53a048aedb560d4971b44705405e75fe599d3946baf503c035c2679b"}
Jul 10 19:11:16.435872 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.435828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:16.438286 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.438259    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:16.711301 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.711272    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:16.847140 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:16.847118    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:11:16.850010 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:11:16.849982    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc773b2eb_474a_4cd6_98e2_5125175eb3f1.slice/crio-a37d512b41957cf9c05f3113a3f5ce04597b1e08aebef77f990d79fb97df952a WatchSource:0}: Error finding container a37d512b41957cf9c05f3113a3f5ce04597b1e08aebef77f990d79fb97df952a: Status 404 returned error can't find the container with id a37d512b41957cf9c05f3113a3f5ce04597b1e08aebef77f990d79fb97df952a
Jul 10 19:11:17.344547 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:17.344514    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerStarted","Data":"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"}
Jul 10 19:11:17.344729 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:17.344582    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerStarted","Data":"a37d512b41957cf9c05f3113a3f5ce04597b1e08aebef77f990d79fb97df952a"}
Jul 10 19:11:18.028160 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:18.028104    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:11:18.028573 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:18.028476    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:20.360719 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:20.360684    2570 generic.go:356] "Generic (PLEG): container finished" podID="79188647-e6db-4318-8c6a-405ae2a43101" containerID="b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50" exitCode=0
Jul 10 19:11:20.361123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:20.360747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerDied","Data":"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"}
Jul 10 19:11:20.362106 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:20.362089    2570 generic.go:356] "Generic (PLEG): container finished" podID="168d15a7-e849-4968-bb60-e19259c523a3" containerID="d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6" exitCode=0
Jul 10 19:11:20.362195 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:20.362147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerDied","Data":"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"}
Jul 10 19:11:21.020550 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.020444    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:21.368571 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.368485    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerStarted","Data":"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"}
Jul 10 19:11:21.368571 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.368547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerStarted","Data":"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"}
Jul 10 19:11:21.369078 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.368863    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:21.369078 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.369031    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:21.370074 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.370052    2570 generic.go:356] "Generic (PLEG): container finished" podID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerID="20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820" exitCode=0
Jul 10 19:11:21.370178 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.370079    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerDied","Data":"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"}
Jul 10 19:11:21.370464 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.370444    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:11:21.371329 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.371314    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:11:21.387587 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:21.387541    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podStartSLOduration=2.4506104779999998 podStartE2EDuration="6.387525488s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.93691501s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:21.386112 +0000 UTC m=+765.520042704" watchObservedRunningTime="2026-07-10 19:11:21.387525488 +0000 UTC m=+765.521456153"
Jul 10 19:11:22.375026 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:22.374984    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:11:26.020666 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:26.020617    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:26.021122 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:26.020801    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:27.379356 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:27.379327    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:11:27.379894 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:27.379871    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:11:28.028158 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:28.028110    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:11:28.028470 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:28.028440    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:31.020728 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:31.020677    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:36.021003 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:36.020945    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:37.379878 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:37.379825    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:11:38.028130 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:38.028080    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 10 19:11:38.028320 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:38.028253    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:38.028436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:38.028408    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:11:38.028554 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:38.028533    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:41.020541 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:41.020494    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 10 19:11:44.478429 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:44.478391    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerStarted","Data":"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"}
Jul 10 19:11:44.478815 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:44.478458    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerStarted","Data":"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"}
Jul 10 19:11:44.478815 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:44.478644    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:44.499172 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:44.499120    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podStartSLOduration=3.19155488 podStartE2EDuration="29.499103718s" totalImagesPullingTime="22.511693391s" totalInitContainerRuntime="3.795855447s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:44.497690799 +0000 UTC m=+788.631621505" watchObservedRunningTime="2026-07-10 19:11:44.499103718 +0000 UTC m=+788.633034385"
Jul 10 19:11:45.483054 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.483020    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:45.484259 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.484220    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:45.636916 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.636883    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:45.794251 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794150    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config\") pod \"79188647-e6db-4318-8c6a-405ae2a43101\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") "
Jul 10 19:11:45.794251 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794196    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") pod \"79188647-e6db-4318-8c6a-405ae2a43101\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") "
Jul 10 19:11:45.794251 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794224    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jftzx\" (UniqueName: \"kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx\") pod \"79188647-e6db-4318-8c6a-405ae2a43101\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") "
Jul 10 19:11:45.794531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794258    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location\") pod \"79188647-e6db-4318-8c6a-405ae2a43101\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") "
Jul 10 19:11:45.794531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794429    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert\") pod \"79188647-e6db-4318-8c6a-405ae2a43101\" (UID: \"79188647-e6db-4318-8c6a-405ae2a43101\") "
Jul 10 19:11:45.794629 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794573    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79188647-e6db-4318-8c6a-405ae2a43101" (UID: "79188647-e6db-4318-8c6a-405ae2a43101"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:11:45.794629 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794601    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79188647-e6db-4318-8c6a-405ae2a43101" (UID: "79188647-e6db-4318-8c6a-405ae2a43101"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:45.794783 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794744    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79188647-e6db-4318-8c6a-405ae2a43101" (UID: "79188647-e6db-4318-8c6a-405ae2a43101"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:45.794852 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794836    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:11:45.794894 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.794856    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79188647-e6db-4318-8c6a-405ae2a43101-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:11:45.796438 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.796419    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79188647-e6db-4318-8c6a-405ae2a43101" (UID: "79188647-e6db-4318-8c6a-405ae2a43101"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:45.796515 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.796501    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx" (OuterVolumeSpecName: "kube-api-access-jftzx") pod "79188647-e6db-4318-8c6a-405ae2a43101" (UID: "79188647-e6db-4318-8c6a-405ae2a43101"). InnerVolumeSpecName "kube-api-access-jftzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:11:45.896150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.896112    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79188647-e6db-4318-8c6a-405ae2a43101-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:11:45.896150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.896143    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jftzx\" (UniqueName: \"kubernetes.io/projected/79188647-e6db-4318-8c6a-405ae2a43101-kube-api-access-jftzx\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:11:45.896150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:45.896153    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79188647-e6db-4318-8c6a-405ae2a43101-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:11:46.489318 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.489286    2570 generic.go:356] "Generic (PLEG): container finished" podID="79188647-e6db-4318-8c6a-405ae2a43101" containerID="6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599" exitCode=0
Jul 10 19:11:46.489704 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.489411    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"
Jul 10 19:11:46.489704 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.489424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerDied","Data":"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"}
Jul 10 19:11:46.489704 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.489489    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8" event={"ID":"79188647-e6db-4318-8c6a-405ae2a43101","Type":"ContainerDied","Data":"07efb80070efcc5836105b45f8760a4a272b4ef8a7151fc07073704ed4ae617d"}
Jul 10 19:11:46.489704 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.489503    2570 scope.go:117] "RemoveContainer" containerID="6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"
Jul 10 19:11:46.490211 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.490185    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:46.498188 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.498070    2570 scope.go:117] "RemoveContainer" containerID="30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"
Jul 10 19:11:46.505951 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.505927    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:11:46.506306 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.506285    2570 scope.go:117] "RemoveContainer" containerID="b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"
Jul 10 19:11:46.508901 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.508880    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cc766-predictor-6fcdcdf9d8-xdjx8"]
Jul 10 19:11:46.514306 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.514283    2570 scope.go:117] "RemoveContainer" containerID="28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"
Jul 10 19:11:46.522204 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522186    2570 scope.go:117] "RemoveContainer" containerID="6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"
Jul 10 19:11:46.522429 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:46.522408    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599\": container with ID starting with 6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599 not found: ID does not exist" containerID="6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"
Jul 10 19:11:46.522498 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522443    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599"} err="failed to get container status \"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599\": rpc error: code = NotFound desc = could not find container \"6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599\": container with ID starting with 6bef066ebed9e0de7c32a3a89b3de738638880e0578cdca33d6f16be008df599 not found: ID does not exist"
Jul 10 19:11:46.522498 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522485    2570 scope.go:117] "RemoveContainer" containerID="30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"
Jul 10 19:11:46.522701 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:46.522682    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb\": container with ID starting with 30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb not found: ID does not exist" containerID="30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"
Jul 10 19:11:46.522795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522705    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb"} err="failed to get container status \"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb\": rpc error: code = NotFound desc = could not find container \"30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb\": container with ID starting with 30419588e5738d5f65690beec35ba349746f729dbe26a53c4594af87f37accfb not found: ID does not exist"
Jul 10 19:11:46.522795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522721    2570 scope.go:117] "RemoveContainer" containerID="b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"
Jul 10 19:11:46.522943 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:46.522922    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50\": container with ID starting with b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50 not found: ID does not exist" containerID="b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"
Jul 10 19:11:46.522996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522953    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50"} err="failed to get container status \"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50\": rpc error: code = NotFound desc = could not find container \"b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50\": container with ID starting with b989f99031f6b4559b79f6534db30aab0610d16cdb4bf8dfe67518d83fdf8e50 not found: ID does not exist"
Jul 10 19:11:46.522996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.522974    2570 scope.go:117] "RemoveContainer" containerID="28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"
Jul 10 19:11:46.523223 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:11:46.523203    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a\": container with ID starting with 28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a not found: ID does not exist" containerID="28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"
Jul 10 19:11:46.523280 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:46.523229    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a"} err="failed to get container status \"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a\": rpc error: code = NotFound desc = could not find container \"28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a\": container with ID starting with 28aced9a5b361052284ea680b5f48200d211ed0ae508af5d2266c6e4fd27b49a not found: ID does not exist"
Jul 10 19:11:47.380448 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:47.380411    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:11:48.414330 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:48.414299    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79188647-e6db-4318-8c6a-405ae2a43101" path="/var/lib/kubelet/pods/79188647-e6db-4318-8c6a-405ae2a43101/volumes"
Jul 10 19:11:51.494133 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:51.494103    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:11:51.494608 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:51.494582    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:11:57.380450 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:11:57.380407    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:12:01.495216 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:01.495134    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:07.380146 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:07.380106    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:12:11.494551 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:11.494511    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:17.380069 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:17.380027    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:12:21.495422 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:21.495377    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:27.379812 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:27.379754    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:12:28.409727 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:28.409678    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:12:31.495032 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:31.494991    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:38.413942 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:38.413915    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:12:41.495534 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:41.495488    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 10 19:12:51.494942 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:51.494911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:12:55.667171 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667134    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667462    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667472    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667485    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="storage-initializer"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667491    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="storage-initializer"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667504    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667510    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667517    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy"
Jul 10 19:12:55.667531 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667521    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy"
Jul 10 19:12:55.667796 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667569    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kserve-container"
Jul 10 19:12:55.667796 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667577    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="agent"
Jul 10 19:12:55.667796 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.667583    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="79188647-e6db-4318-8c6a-405ae2a43101" containerName="kube-rbac-proxy"
Jul 10 19:12:55.670535 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.670517    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.672987 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.672967    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e3394-serving-cert\""
Jul 10 19:12:55.673090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.673003    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e3394-kube-rbac-proxy-sar-config\""
Jul 10 19:12:55.679322 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.679298    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:12:55.771801 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.771730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.771967 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.771818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.872487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.872453    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.872688 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.872510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.873118 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.873096    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.874885 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.874863    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls\") pod \"model-chainer-raw-e3394-5c795895db-9tdkc\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") " pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:55.981016 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:55.980922    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:56.112284 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:56.112249    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:12:56.115479 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:12:56.115451    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a112353_8b4b_4343_b702_6c60ac82c551.slice/crio-cbc4ec073a7e7c4c6dcb9effcc46bbcf04516b66cedcdb14d7f12608de1dfdee WatchSource:0}: Error finding container cbc4ec073a7e7c4c6dcb9effcc46bbcf04516b66cedcdb14d7f12608de1dfdee: Status 404 returned error can't find the container with id cbc4ec073a7e7c4c6dcb9effcc46bbcf04516b66cedcdb14d7f12608de1dfdee
Jul 10 19:12:56.782877 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:56.782841    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" event={"ID":"2a112353-8b4b-4343-b702-6c60ac82c551","Type":"ContainerStarted","Data":"cbc4ec073a7e7c4c6dcb9effcc46bbcf04516b66cedcdb14d7f12608de1dfdee"}
Jul 10 19:12:58.792705 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:58.792668    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" event={"ID":"2a112353-8b4b-4343-b702-6c60ac82c551","Type":"ContainerStarted","Data":"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"}
Jul 10 19:12:58.793117 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:58.792805    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:12:58.810433 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:12:58.810383    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podStartSLOduration=1.580242052 podStartE2EDuration="3.810368597s" totalImagesPullingTime="2.230126545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:58.809555956 +0000 UTC m=+862.943486625" watchObservedRunningTime="2026-07-10 19:12:58.810368597 +0000 UTC m=+862.944299262"
Jul 10 19:13:04.802341 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:04.802309    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:13:05.693185 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:05.693149    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:13:05.693389 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:05.693344    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" containerID="cri-o://695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc" gracePeriod=30
Jul 10 19:13:05.934838 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:05.934805    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:13:05.935253 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:05.935053    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" containerID="cri-o://6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c" gracePeriod=30
Jul 10 19:13:05.935253 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:05.935100    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kube-rbac-proxy" containerID="cri-o://f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265" gracePeriod=30
Jul 10 19:13:06.044998 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.044968    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:13:06.047649 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.047625    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.050545 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.050521    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-fed9d-kube-rbac-proxy-sar-config\""
Jul 10 19:13:06.050666 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.050541    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-fed9d-predictor-serving-cert\""
Jul 10 19:13:06.060885 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.060857    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:13:06.138371 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.138333    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:13:06.141911 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.141885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.142047 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.141891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.142047 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.142009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdbb\" (UniqueName: \"kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.142150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.142054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.142150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.142113    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.142224 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.142170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.144378 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.144360    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-fed9d-predictor-serving-cert\""
Jul 10 19:13:06.144503 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.144380    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-fed9d-kube-rbac-proxy-sar-config\""
Jul 10 19:13:06.152082 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.152064    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:13:06.242699 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.242603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.242699 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.242652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.242699 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.242681    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.243018 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.242705    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.242790    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdbb\" (UniqueName: \"kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243358    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtwn\" (UniqueName: \"kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243517    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.244141    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.243969    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.244745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.244458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.247746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.247722    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.257564 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.257542    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:13:06.257840 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.257812    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container" containerID="cri-o://b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c" gracePeriod=30
Jul 10 19:13:06.257940 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.257836    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kube-rbac-proxy" containerID="cri-o://1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1" gracePeriod=30
Jul 10 19:13:06.258360 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.258334    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdbb\" (UniqueName: \"kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb\") pod \"isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.344301 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtwn\" (UniqueName: \"kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.344496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.344496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.344496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344395    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.344496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.344957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.344924    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.345142 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.345113    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.345257 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.345160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.346926 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.346905    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.352707 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.352650    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtwn\" (UniqueName: \"kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn\") pod \"isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.358326 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.358307    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:06.452496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.452445    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:06.490170 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.490125    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 10 19:13:06.497238 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.497212    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:13:06.499884 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:13:06.499856    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7201858_6980_453d_8745_c8ff65e9a191.slice/crio-ac6b14c7499fc2e1fb461754b5b42ad2dcf56100860b109ae8d82a756232d26e WatchSource:0}: Error finding container ac6b14c7499fc2e1fb461754b5b42ad2dcf56100860b109ae8d82a756232d26e: Status 404 returned error can't find the container with id ac6b14c7499fc2e1fb461754b5b42ad2dcf56100860b109ae8d82a756232d26e
Jul 10 19:13:06.594055 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.593990    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:13:06.596711 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:13:06.596678    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae4fa15_4c11_4944_a7c0_f17bac5effaa.slice/crio-6014bcb00f394741db523bc82d71bfd57f2fec499b9e820193812f20a8e7d518 WatchSource:0}: Error finding container 6014bcb00f394741db523bc82d71bfd57f2fec499b9e820193812f20a8e7d518: Status 404 returned error can't find the container with id 6014bcb00f394741db523bc82d71bfd57f2fec499b9e820193812f20a8e7d518
Jul 10 19:13:06.827893 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.827794    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerStarted","Data":"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"}
Jul 10 19:13:06.827893 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.827855    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerStarted","Data":"ac6b14c7499fc2e1fb461754b5b42ad2dcf56100860b109ae8d82a756232d26e"}
Jul 10 19:13:06.830149 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.830116    2570 generic.go:356] "Generic (PLEG): container finished" podID="168d15a7-e849-4968-bb60-e19259c523a3" containerID="f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265" exitCode=2
Jul 10 19:13:06.830294 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.830186    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerDied","Data":"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"}
Jul 10 19:13:06.832492 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.832464    2570 generic.go:356] "Generic (PLEG): container finished" podID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerID="1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1" exitCode=2
Jul 10 19:13:06.832599 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.832531    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerDied","Data":"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"}
Jul 10 19:13:06.837882 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.837853    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerStarted","Data":"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"}
Jul 10 19:13:06.838022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:06.837910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerStarted","Data":"6014bcb00f394741db523bc82d71bfd57f2fec499b9e820193812f20a8e7d518"}
Jul 10 19:13:07.376139 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:07.376097    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 10 19:13:08.409626 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:08.409581    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 10 19:13:09.800892 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:09.800850    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:10.104744 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.104719    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:13:10.178343 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178313    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location\") pod \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") "
Jul 10 19:13:10.178530 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178354    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") pod \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") "
Jul 10 19:13:10.178530 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178389    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config\") pod \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") "
Jul 10 19:13:10.178530 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178404    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rcqm\" (UniqueName: \"kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm\") pod \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") "
Jul 10 19:13:10.178530 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178435    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert\") pod \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\" (UID: \"c773b2eb-474a-4cd6-98e2-5125175eb3f1\") "
Jul 10 19:13:10.178736 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178685    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c773b2eb-474a-4cd6-98e2-5125175eb3f1" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:13:10.178850 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178826    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c773b2eb-474a-4cd6-98e2-5125175eb3f1" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:10.178898 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.178856    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c773b2eb-474a-4cd6-98e2-5125175eb3f1" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:10.180510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.180487    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm" (OuterVolumeSpecName: "kube-api-access-2rcqm") pod "c773b2eb-474a-4cd6-98e2-5125175eb3f1" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1"). InnerVolumeSpecName "kube-api-access-2rcqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:13:10.180599 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.180493    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c773b2eb-474a-4cd6-98e2-5125175eb3f1" (UID: "c773b2eb-474a-4cd6-98e2-5125175eb3f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:13:10.279552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.279507    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:10.279552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.279545    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:10.279552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.279557    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c773b2eb-474a-4cd6-98e2-5125175eb3f1-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:10.279813 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.279567    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:10.279813 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.279576    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rcqm\" (UniqueName: \"kubernetes.io/projected/c773b2eb-474a-4cd6-98e2-5125175eb3f1-kube-api-access-2rcqm\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:10.857745 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.857711    2570 generic.go:356] "Generic (PLEG): container finished" podID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerID="b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c" exitCode=0
Jul 10 19:13:10.858150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.857789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerDied","Data":"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"}
Jul 10 19:13:10.858150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.857853    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb" event={"ID":"c773b2eb-474a-4cd6-98e2-5125175eb3f1","Type":"ContainerDied","Data":"a37d512b41957cf9c05f3113a3f5ce04597b1e08aebef77f990d79fb97df952a"}
Jul 10 19:13:10.858150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.857803    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"
Jul 10 19:13:10.858150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.857878    2570 scope.go:117] "RemoveContainer" containerID="1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"
Jul 10 19:13:10.859328 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.859311    2570 generic.go:356] "Generic (PLEG): container finished" podID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerID="2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497" exitCode=0
Jul 10 19:13:10.859420 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.859383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerDied","Data":"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"}
Jul 10 19:13:10.860933 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.860916    2570 generic.go:356] "Generic (PLEG): container finished" podID="e7201858-6980-453d-8745-c8ff65e9a191" containerID="1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835" exitCode=0
Jul 10 19:13:10.861018 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.860952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerDied","Data":"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"}
Jul 10 19:13:10.908337 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.908303    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:13:10.912301 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.912270    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3394-predictor-9c59fc957-9vgzb"]
Jul 10 19:13:10.960559 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.960506    2570 scope.go:117] "RemoveContainer" containerID="b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"
Jul 10 19:13:10.979614 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.979247    2570 scope.go:117] "RemoveContainer" containerID="20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"
Jul 10 19:13:10.992361 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.992335    2570 scope.go:117] "RemoveContainer" containerID="1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"
Jul 10 19:13:10.992784 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:10.992742    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1\": container with ID starting with 1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1 not found: ID does not exist" containerID="1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"
Jul 10 19:13:10.992861 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.992819    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1"} err="failed to get container status \"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1\": rpc error: code = NotFound desc = could not find container \"1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1\": container with ID starting with 1c5c5e0a557c0d40b21f9af6246f7588baae77b8a510f1c1a1cfb96418eac5c1 not found: ID does not exist"
Jul 10 19:13:10.992861 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.992847    2570 scope.go:117] "RemoveContainer" containerID="b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"
Jul 10 19:13:10.993153 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:10.993129    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c\": container with ID starting with b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c not found: ID does not exist" containerID="b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"
Jul 10 19:13:10.993215 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.993165    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c"} err="failed to get container status \"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c\": rpc error: code = NotFound desc = could not find container \"b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c\": container with ID starting with b67acb746b721fb88d781b4515ef5c38c6a280c07f136010f1a4952e755a0b2c not found: ID does not exist"
Jul 10 19:13:10.993215 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.993190    2570 scope.go:117] "RemoveContainer" containerID="20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"
Jul 10 19:13:10.993477 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:10.993452    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820\": container with ID starting with 20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820 not found: ID does not exist" containerID="20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"
Jul 10 19:13:10.993532 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:10.993487    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820"} err="failed to get container status \"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820\": rpc error: code = NotFound desc = could not find container \"20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820\": container with ID starting with 20719de1c433057f5cdb678f7793ad2c51973ebd00735631554ca5c015ea3820 not found: ID does not exist"
Jul 10 19:13:11.108914 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.108889    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:13:11.187074 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location\") pod \"168d15a7-e849-4968-bb60-e19259c523a3\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") "
Jul 10 19:13:11.187228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187089    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp2hg\" (UniqueName: \"kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg\") pod \"168d15a7-e849-4968-bb60-e19259c523a3\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") "
Jul 10 19:13:11.187228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187117    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert\") pod \"168d15a7-e849-4968-bb60-e19259c523a3\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") "
Jul 10 19:13:11.187228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187141    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls\") pod \"168d15a7-e849-4968-bb60-e19259c523a3\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") "
Jul 10 19:13:11.187228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187165    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config\") pod \"168d15a7-e849-4968-bb60-e19259c523a3\" (UID: \"168d15a7-e849-4968-bb60-e19259c523a3\") "
Jul 10 19:13:11.187456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187364    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "168d15a7-e849-4968-bb60-e19259c523a3" (UID: "168d15a7-e849-4968-bb60-e19259c523a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:13:11.187456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187442    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "168d15a7-e849-4968-bb60-e19259c523a3" (UID: "168d15a7-e849-4968-bb60-e19259c523a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:11.187590 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.187558    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "168d15a7-e849-4968-bb60-e19259c523a3" (UID: "168d15a7-e849-4968-bb60-e19259c523a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:11.189170 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.189147    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg" (OuterVolumeSpecName: "kube-api-access-wp2hg") pod "168d15a7-e849-4968-bb60-e19259c523a3" (UID: "168d15a7-e849-4968-bb60-e19259c523a3"). InnerVolumeSpecName "kube-api-access-wp2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:13:11.189257 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.189239    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "168d15a7-e849-4968-bb60-e19259c523a3" (UID: "168d15a7-e849-4968-bb60-e19259c523a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:13:11.288091 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.288051    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168d15a7-e849-4968-bb60-e19259c523a3-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:11.288091 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.288084    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2hg\" (UniqueName: \"kubernetes.io/projected/168d15a7-e849-4968-bb60-e19259c523a3-kube-api-access-wp2hg\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:11.288091 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.288094    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:11.288091 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.288103    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168d15a7-e849-4968-bb60-e19259c523a3-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:11.288386 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.288111    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168d15a7-e849-4968-bb60-e19259c523a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:11.867357 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.867320    2570 generic.go:356] "Generic (PLEG): container finished" podID="168d15a7-e849-4968-bb60-e19259c523a3" containerID="6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c" exitCode=0
Jul 10 19:13:11.867795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.867386    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"
Jul 10 19:13:11.867795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.867413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerDied","Data":"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"}
Jul 10 19:13:11.867795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.867486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf" event={"ID":"168d15a7-e849-4968-bb60-e19259c523a3","Type":"ContainerDied","Data":"d7cac34f53a048aedb560d4971b44705405e75fe599d3946baf503c035c2679b"}
Jul 10 19:13:11.867795 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.867524    2570 scope.go:117] "RemoveContainer" containerID="f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"
Jul 10 19:13:11.870970 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.870938    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerStarted","Data":"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"}
Jul 10 19:13:11.871098 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.871014    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerStarted","Data":"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"}
Jul 10 19:13:11.871353 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.871298    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:11.873402 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.873375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerStarted","Data":"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"}
Jul 10 19:13:11.873510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.873438    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerStarted","Data":"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"}
Jul 10 19:13:11.873717 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.873699    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:11.876733 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.876712    2570 scope.go:117] "RemoveContainer" containerID="6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"
Jul 10 19:13:11.884443 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.884426    2570 scope.go:117] "RemoveContainer" containerID="d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"
Jul 10 19:13:11.891919 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.891881    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podStartSLOduration=2.009725535 podStartE2EDuration="5.891864333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882138798s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:11.890308508 +0000 UTC m=+876.024239175" watchObservedRunningTime="2026-07-10 19:13:11.891864333 +0000 UTC m=+876.025794992"
Jul 10 19:13:11.891996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.891901    2570 scope.go:117] "RemoveContainer" containerID="f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"
Jul 10 19:13:11.892319 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:11.892297    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265\": container with ID starting with f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265 not found: ID does not exist" containerID="f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"
Jul 10 19:13:11.892400 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.892330    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265"} err="failed to get container status \"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265\": rpc error: code = NotFound desc = could not find container \"f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265\": container with ID starting with f786ec2f73f05d9e9d8ebdd1a0028805379241ef369d2f67a629b067f7777265 not found: ID does not exist"
Jul 10 19:13:11.892400 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.892356    2570 scope.go:117] "RemoveContainer" containerID="6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"
Jul 10 19:13:11.892746 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:11.892725    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c\": container with ID starting with 6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c not found: ID does not exist" containerID="6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"
Jul 10 19:13:11.892927 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.892751    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c"} err="failed to get container status \"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c\": rpc error: code = NotFound desc = could not find container \"6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c\": container with ID starting with 6b651e7f6702ef706927fc7668c48d90b1b7f84e7b1055a3e5250e4c2862052c not found: ID does not exist"
Jul 10 19:13:11.892927 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.892791    2570 scope.go:117] "RemoveContainer" containerID="d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"
Jul 10 19:13:11.893120 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:11.893097    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6\": container with ID starting with d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6 not found: ID does not exist" containerID="d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"
Jul 10 19:13:11.893184 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.893140    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6"} err="failed to get container status \"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6\": rpc error: code = NotFound desc = could not find container \"d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6\": container with ID starting with d6cfde5a28cdba2ece633ebfb94cc82068a9eee4a42ee88a7fc9bd840edc4ba6 not found: ID does not exist"
Jul 10 19:13:11.903903 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.903877    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:13:11.908057 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.908036    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3394-predictor-6db968d9b4-drbvf"]
Jul 10 19:13:11.927387 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:11.927346    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podStartSLOduration=1.96904295 podStartE2EDuration="5.927333689s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.958290739s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:11.92651899 +0000 UTC m=+876.060449678" watchObservedRunningTime="2026-07-10 19:13:11.927333689 +0000 UTC m=+876.061264356"
Jul 10 19:13:12.415102 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.415066    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="168d15a7-e849-4968-bb60-e19259c523a3" path="/var/lib/kubelet/pods/168d15a7-e849-4968-bb60-e19259c523a3/volumes"
Jul 10 19:13:12.415555 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.415542    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" path="/var/lib/kubelet/pods/c773b2eb-474a-4cd6-98e2-5125175eb3f1/volumes"
Jul 10 19:13:12.879920 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.879829    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:12.879920 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.879893    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:12.880871 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.880845    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:12.880967 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:12.880917    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:13.883809 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:13.883743    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:13.884166 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:13.883812    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:14.800603 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:14.800556    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:18.888136 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:18.888104    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:13:18.888549 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:18.888495    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:13:18.888624 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:18.888493    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:18.889040 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:18.889013    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:19.801092 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:19.801050    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:19.801246 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:19.801149    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:13:24.801397 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:24.801355    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:28.889633 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:28.889547    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:28.890088 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:28.889547    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:29.801085 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:29.801040    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:34.801271 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:34.801230    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:13:35.853532 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.853505    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:13:35.976827 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.976732    2570 generic.go:356] "Generic (PLEG): container finished" podID="2a112353-8b4b-4343-b702-6c60ac82c551" containerID="695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc" exitCode=0
Jul 10 19:13:35.976827 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.976788    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" event={"ID":"2a112353-8b4b-4343-b702-6c60ac82c551","Type":"ContainerDied","Data":"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"}
Jul 10 19:13:35.977080 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.976802    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"
Jul 10 19:13:35.977080 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.976851    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc" event={"ID":"2a112353-8b4b-4343-b702-6c60ac82c551","Type":"ContainerDied","Data":"cbc4ec073a7e7c4c6dcb9effcc46bbcf04516b66cedcdb14d7f12608de1dfdee"}
Jul 10 19:13:35.977080 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.976875    2570 scope.go:117] "RemoveContainer" containerID="695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"
Jul 10 19:13:35.984832 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.984813    2570 scope.go:117] "RemoveContainer" containerID="695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"
Jul 10 19:13:35.985113 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:13:35.985092    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc\": container with ID starting with 695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc not found: ID does not exist" containerID="695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"
Jul 10 19:13:35.985150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.985126    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc"} err="failed to get container status \"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc\": rpc error: code = NotFound desc = could not find container \"695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc\": container with ID starting with 695c08beaeebd3cd7c34f0e29b348bf6292b2b55dffd546d2e7f6169d740effc not found: ID does not exist"
Jul 10 19:13:35.994423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.994394    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle\") pod \"2a112353-8b4b-4343-b702-6c60ac82c551\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") "
Jul 10 19:13:35.994539 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.994451    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls\") pod \"2a112353-8b4b-4343-b702-6c60ac82c551\" (UID: \"2a112353-8b4b-4343-b702-6c60ac82c551\") "
Jul 10 19:13:35.994731 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.994708    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2a112353-8b4b-4343-b702-6c60ac82c551" (UID: "2a112353-8b4b-4343-b702-6c60ac82c551"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:35.996456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:35.996434    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a112353-8b4b-4343-b702-6c60ac82c551" (UID: "2a112353-8b4b-4343-b702-6c60ac82c551"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:13:36.095552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.095512    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a112353-8b4b-4343-b702-6c60ac82c551-openshift-service-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:36.095552 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.095542    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a112353-8b4b-4343-b702-6c60ac82c551-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:13:36.302169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.302135    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:13:36.303424 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.303402    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e3394-5c795895db-9tdkc"]
Jul 10 19:13:36.335883 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.335857    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:13:36.336187 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.336167    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:13:36.414651 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:36.414621    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" path="/var/lib/kubelet/pods/2a112353-8b4b-4343-b702-6c60ac82c551/volumes"
Jul 10 19:13:38.888990 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:38.888938    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:38.889434 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:38.888965    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:48.888575 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:48.888529    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:48.889076 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:48.889046    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:13:58.889030 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:58.888986    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:13:58.889030 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:13:58.889021    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:14:08.888616 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:08.888572    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:08.889152 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:08.888956    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:14:18.888965 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:18.888929    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:14:18.891282 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:18.888980    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:14:28.889879 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:28.889851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:14:45.989603 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989568    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989903    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="storage-initializer"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989915    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="storage-initializer"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989927    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989935    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989943    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989948    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989954    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="storage-initializer"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989959    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="storage-initializer"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989973    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989980    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989987    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.989992    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990000    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990004    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990066    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kserve-container"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990075    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="168d15a7-e849-4968-bb60-e19259c523a3" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990082    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a112353-8b4b-4343-b702-6c60ac82c551" containerName="model-chainer-raw-e3394"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990088    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kube-rbac-proxy"
Jul 10 19:14:45.990090 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.990095    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c773b2eb-474a-4cd6-98e2-5125175eb3f1" containerName="kserve-container"
Jul 10 19:14:45.993004 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.992988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:45.995473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.995444    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-fed9d-serving-cert\""
Jul 10 19:14:45.995473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:45.995462    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-fed9d-kube-rbac-proxy-sar-config\""
Jul 10 19:14:46.001013 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.000994    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:14:46.061943 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.061901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.062110 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.062036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.162517 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.162472    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.162693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.162526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.162693 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:14:46.162631    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-serving-cert: secret "model-chainer-raw-hpa-fed9d-serving-cert" not found
Jul 10 19:14:46.162796 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:14:46.162717    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls podName:9805083d-e472-4d19-a017-3bcc58721940 nodeName:}" failed. No retries permitted until 2026-07-10 19:14:46.662698415 +0000 UTC m=+970.796629063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls") pod "model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" (UID: "9805083d-e472-4d19-a017-3bcc58721940") : secret "model-chainer-raw-hpa-fed9d-serving-cert" not found
Jul 10 19:14:46.163189 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.163173    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.667335 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.667301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.669583 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.669551    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") pod \"model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:46.903409 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:46.903367    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:47.034511 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:47.034487    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:14:47.035876 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:14:47.035848    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9805083d_e472_4d19_a017_3bcc58721940.slice/crio-5fe2ff7d81f57a9de2311a3429637c4fd0ce684c2c5a82e47f9a9cbfa0e7de4f WatchSource:0}: Error finding container 5fe2ff7d81f57a9de2311a3429637c4fd0ce684c2c5a82e47f9a9cbfa0e7de4f: Status 404 returned error can't find the container with id 5fe2ff7d81f57a9de2311a3429637c4fd0ce684c2c5a82e47f9a9cbfa0e7de4f
Jul 10 19:14:47.269787 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:47.269676    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" event={"ID":"9805083d-e472-4d19-a017-3bcc58721940","Type":"ContainerStarted","Data":"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"}
Jul 10 19:14:47.269787 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:47.269755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" event={"ID":"9805083d-e472-4d19-a017-3bcc58721940","Type":"ContainerStarted","Data":"5fe2ff7d81f57a9de2311a3429637c4fd0ce684c2c5a82e47f9a9cbfa0e7de4f"}
Jul 10 19:14:47.269971 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:47.269820    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:47.288330 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:47.288282    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podStartSLOduration=2.28826179 podStartE2EDuration="2.28826179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:14:47.287203085 +0000 UTC m=+971.421133752" watchObservedRunningTime="2026-07-10 19:14:47.28826179 +0000 UTC m=+971.422192458"
Jul 10 19:14:53.280057 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:53.280028    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:14:56.050088 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.050049    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:14:56.050473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.050280    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" containerID="cri-o://ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0" gracePeriod=30
Jul 10 19:14:56.253171 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.253137    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:14:56.253393 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.253367    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" containerID="cri-o://0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14" gracePeriod=30
Jul 10 19:14:56.253469 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.253441    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kube-rbac-proxy" containerID="cri-o://2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863" gracePeriod=30
Jul 10 19:14:56.282887 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.282856    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:14:56.286473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.286451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.289101 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.289079    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-358e7-predictor-serving-cert\""
Jul 10 19:14:56.289304 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.289281    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-358e7-kube-rbac-proxy-sar-config\""
Jul 10 19:14:56.298445 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.298420    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:14:56.400104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.400077    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:14:56.400370 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.400322    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" containerID="cri-o://39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650" gracePeriod=30
Jul 10 19:14:56.400455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.400351    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kube-rbac-proxy" containerID="cri-o://faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797" gracePeriod=30
Jul 10 19:14:56.450585 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.450550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.450746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.450596    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.450746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.450620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmhb\" (UniqueName: \"kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.551969 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.551934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.552144 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.552018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmhb\" (UniqueName: \"kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.552144 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.552120    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.552630 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.552598    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.554729 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.554706    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.560238 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.560209    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmhb\" (UniqueName: \"kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb\") pod \"message-dumper-raw-358e7-predictor-9f5c55544-9gxsq\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") " pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.596637 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.596609    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:56.730600 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:56.730572    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:14:56.733200 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:14:56.733157    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb26912_ed75_416d_814c_1de8d6d5f0c1.slice/crio-4c180a061bd38eb1c33920f589f5748c46692decf884cfbfd05c054bf925e793 WatchSource:0}: Error finding container 4c180a061bd38eb1c33920f589f5748c46692decf884cfbfd05c054bf925e793: Status 404 returned error can't find the container with id 4c180a061bd38eb1c33920f589f5748c46692decf884cfbfd05c054bf925e793
Jul 10 19:14:57.314294 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:57.314261    2570 generic.go:356] "Generic (PLEG): container finished" podID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerID="faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797" exitCode=2
Jul 10 19:14:57.314693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:57.314331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerDied","Data":"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"}
Jul 10 19:14:57.316368 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:57.316346    2570 generic.go:356] "Generic (PLEG): container finished" podID="e7201858-6980-453d-8745-c8ff65e9a191" containerID="2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863" exitCode=2
Jul 10 19:14:57.316473 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:57.316418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerDied","Data":"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"}
Jul 10 19:14:57.317472 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:57.317456    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerStarted","Data":"4c180a061bd38eb1c33920f589f5748c46692decf884cfbfd05c054bf925e793"}
Jul 10 19:14:58.278631 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.278594    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:14:58.328591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.328553    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerStarted","Data":"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"}
Jul 10 19:14:58.329074 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.328642    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerStarted","Data":"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"}
Jul 10 19:14:58.329074 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.328678    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:58.329074 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.328796    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:58.330577 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.330549    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:14:58.350445 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.350379    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" podStartSLOduration=1.291283879 podStartE2EDuration="2.350358056s" totalImagesPullingTime="1.059074177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:14:58.348366409 +0000 UTC m=+982.482297075" watchObservedRunningTime="2026-07-10 19:14:58.350358056 +0000 UTC m=+982.484288724"
Jul 10 19:14:58.884151 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.884112    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 10 19:14:58.884332 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.884117    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 10 19:14:58.888720 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.888694    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 10 19:14:58.889909 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:14:58.889890    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 10 19:15:00.244375 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.244338    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:15:00.338965 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.338878    2570 generic.go:356] "Generic (PLEG): container finished" podID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerID="39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650" exitCode=0
Jul 10 19:15:00.339109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.338958    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"
Jul 10 19:15:00.339109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.338974    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerDied","Data":"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"}
Jul 10 19:15:00.339109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.339032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb" event={"ID":"4ae4fa15-4c11-4944-a7c0-f17bac5effaa","Type":"ContainerDied","Data":"6014bcb00f394741db523bc82d71bfd57f2fec499b9e820193812f20a8e7d518"}
Jul 10 19:15:00.339109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.339061    2570 scope.go:117] "RemoveContainer" containerID="faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"
Jul 10 19:15:00.347093 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.347069    2570 scope.go:117] "RemoveContainer" containerID="39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"
Jul 10 19:15:00.354349 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.354330    2570 scope.go:117] "RemoveContainer" containerID="2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"
Jul 10 19:15:00.361538 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.361520    2570 scope.go:117] "RemoveContainer" containerID="faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"
Jul 10 19:15:00.361873 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:00.361849    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797\": container with ID starting with faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797 not found: ID does not exist" containerID="faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"
Jul 10 19:15:00.361931 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.361883    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797"} err="failed to get container status \"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797\": rpc error: code = NotFound desc = could not find container \"faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797\": container with ID starting with faf970df63947920c05118dc5aa71402c37c90ffc6cfcf44c8edd971f0195797 not found: ID does not exist"
Jul 10 19:15:00.361931 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.361903    2570 scope.go:117] "RemoveContainer" containerID="39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"
Jul 10 19:15:00.362134 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:00.362113    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650\": container with ID starting with 39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650 not found: ID does not exist" containerID="39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"
Jul 10 19:15:00.362175 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.362140    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650"} err="failed to get container status \"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650\": rpc error: code = NotFound desc = could not find container \"39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650\": container with ID starting with 39a53a4f491b9bd13d15a49988a19ab70cedd4dc5b6137ecca51b984d3255650 not found: ID does not exist"
Jul 10 19:15:00.362175 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.362157    2570 scope.go:117] "RemoveContainer" containerID="2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"
Jul 10 19:15:00.362415 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:00.362397    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497\": container with ID starting with 2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497 not found: ID does not exist" containerID="2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"
Jul 10 19:15:00.362467 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.362422    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497"} err="failed to get container status \"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497\": rpc error: code = NotFound desc = could not find container \"2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497\": container with ID starting with 2d2c23c575169ec93fb9723ce8569753125c259bad7197e973aec89aa5a2b497 not found: ID does not exist"
Jul 10 19:15:00.384737 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.384708    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls\") pod \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") "
Jul 10 19:15:00.384867 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.384749    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config\") pod \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") "
Jul 10 19:15:00.384867 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.384793    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location\") pod \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") "
Jul 10 19:15:00.384867 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.384820    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgtwn\" (UniqueName: \"kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn\") pod \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") "
Jul 10 19:15:00.384963 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.384868    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert\") pod \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\" (UID: \"4ae4fa15-4c11-4944-a7c0-f17bac5effaa\") "
Jul 10 19:15:00.385109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.385082    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ae4fa15-4c11-4944-a7c0-f17bac5effaa" (UID: "4ae4fa15-4c11-4944-a7c0-f17bac5effaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:15:00.385224 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.385125    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ae4fa15-4c11-4944-a7c0-f17bac5effaa" (UID: "4ae4fa15-4c11-4944-a7c0-f17bac5effaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:00.385322 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.385300    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ae4fa15-4c11-4944-a7c0-f17bac5effaa" (UID: "4ae4fa15-4c11-4944-a7c0-f17bac5effaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:00.386817 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.386798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn" (OuterVolumeSpecName: "kube-api-access-fgtwn") pod "4ae4fa15-4c11-4944-a7c0-f17bac5effaa" (UID: "4ae4fa15-4c11-4944-a7c0-f17bac5effaa"). InnerVolumeSpecName "kube-api-access-fgtwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:15:00.386880 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.386854    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ae4fa15-4c11-4944-a7c0-f17bac5effaa" (UID: "4ae4fa15-4c11-4944-a7c0-f17bac5effaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:00.486262 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.486218    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:00.486262 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.486254    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:00.486262 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.486266    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:00.486499 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.486278    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtwn\" (UniqueName: \"kubernetes.io/projected/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-kube-api-access-fgtwn\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:00.486499 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.486290    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae4fa15-4c11-4944-a7c0-f17bac5effaa-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:00.657105 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.657065    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:15:00.663228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:00.663201    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-fed9d-predictor-55bbcc449-znpdb"]
Jul 10 19:15:01.210203 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.210172    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:15:01.294323 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294239    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls\") pod \"e7201858-6980-453d-8745-c8ff65e9a191\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") "
Jul 10 19:15:01.294323 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294280    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location\") pod \"e7201858-6980-453d-8745-c8ff65e9a191\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") "
Jul 10 19:15:01.294747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294341    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pdbb\" (UniqueName: \"kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb\") pod \"e7201858-6980-453d-8745-c8ff65e9a191\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") "
Jul 10 19:15:01.294747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294370    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config\") pod \"e7201858-6980-453d-8745-c8ff65e9a191\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") "
Jul 10 19:15:01.294747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294397    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert\") pod \"e7201858-6980-453d-8745-c8ff65e9a191\" (UID: \"e7201858-6980-453d-8745-c8ff65e9a191\") "
Jul 10 19:15:01.294747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294669    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7201858-6980-453d-8745-c8ff65e9a191" (UID: "e7201858-6980-453d-8745-c8ff65e9a191"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:15:01.294747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294722    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7201858-6980-453d-8745-c8ff65e9a191" (UID: "e7201858-6980-453d-8745-c8ff65e9a191"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:01.294947 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.294808    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7201858-6980-453d-8745-c8ff65e9a191" (UID: "e7201858-6980-453d-8745-c8ff65e9a191"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:01.296452 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.296430    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7201858-6980-453d-8745-c8ff65e9a191" (UID: "e7201858-6980-453d-8745-c8ff65e9a191"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:01.296557 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.296432    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb" (OuterVolumeSpecName: "kube-api-access-9pdbb") pod "e7201858-6980-453d-8745-c8ff65e9a191" (UID: "e7201858-6980-453d-8745-c8ff65e9a191"). InnerVolumeSpecName "kube-api-access-9pdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:15:01.345996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.345965    2570 generic.go:356] "Generic (PLEG): container finished" podID="e7201858-6980-453d-8745-c8ff65e9a191" containerID="0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14" exitCode=0
Jul 10 19:15:01.346134 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.346015    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerDied","Data":"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"}
Jul 10 19:15:01.346134 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.346023    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"
Jul 10 19:15:01.346134 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.346070    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz" event={"ID":"e7201858-6980-453d-8745-c8ff65e9a191","Type":"ContainerDied","Data":"ac6b14c7499fc2e1fb461754b5b42ad2dcf56100860b109ae8d82a756232d26e"}
Jul 10 19:15:01.346134 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.346099    2570 scope.go:117] "RemoveContainer" containerID="2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"
Jul 10 19:15:01.355236 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.355216    2570 scope.go:117] "RemoveContainer" containerID="0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"
Jul 10 19:15:01.362329 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.362308    2570 scope.go:117] "RemoveContainer" containerID="1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"
Jul 10 19:15:01.368338 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.368315    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:15:01.369966 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.369947    2570 scope.go:117] "RemoveContainer" containerID="2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"
Jul 10 19:15:01.370611 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:01.370328    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863\": container with ID starting with 2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863 not found: ID does not exist" containerID="2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"
Jul 10 19:15:01.370611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.370367    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863"} err="failed to get container status \"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863\": rpc error: code = NotFound desc = could not find container \"2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863\": container with ID starting with 2e2111bef21065d6e9beda425e715c1a418a900725132a4e7baab04e7a17b863 not found: ID does not exist"
Jul 10 19:15:01.370611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.370411    2570 scope.go:117] "RemoveContainer" containerID="0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"
Jul 10 19:15:01.370839 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:01.370712    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14\": container with ID starting with 0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14 not found: ID does not exist" containerID="0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"
Jul 10 19:15:01.370839 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.370744    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14"} err="failed to get container status \"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14\": rpc error: code = NotFound desc = could not find container \"0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14\": container with ID starting with 0d645301f28df400c0254f2c781c5a8430156ba44b9a2626db8cff4c73d18b14 not found: ID does not exist"
Jul 10 19:15:01.370986 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.370966    2570 scope.go:117] "RemoveContainer" containerID="1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"
Jul 10 19:15:01.371241 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:01.371226    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835\": container with ID starting with 1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835 not found: ID does not exist" containerID="1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"
Jul 10 19:15:01.371296 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.371245    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835"} err="failed to get container status \"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835\": rpc error: code = NotFound desc = could not find container \"1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835\": container with ID starting with 1b027cb96cb4feb3ee25d7d23dec5d3dc1ea4cea4f2294ea1fda306ac1aa3835 not found: ID does not exist"
Jul 10 19:15:01.372137 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.372120    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-fed9d-predictor-67ccfccd84-rcdqz"]
Jul 10 19:15:01.395693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.395662    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7201858-6980-453d-8745-c8ff65e9a191-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:01.395693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.395689    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7201858-6980-453d-8745-c8ff65e9a191-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:01.395693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.395700    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdbb\" (UniqueName: \"kubernetes.io/projected/e7201858-6980-453d-8745-c8ff65e9a191-kube-api-access-9pdbb\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:01.395896 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.395710    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:01.395896 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:01.395721    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7201858-6980-453d-8745-c8ff65e9a191-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:02.414263 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:02.414230    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" path="/var/lib/kubelet/pods/4ae4fa15-4c11-4944-a7c0-f17bac5effaa/volumes"
Jul 10 19:15:02.414705 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:02.414691    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7201858-6980-453d-8745-c8ff65e9a191" path="/var/lib/kubelet/pods/e7201858-6980-453d-8745-c8ff65e9a191/volumes"
Jul 10 19:15:03.278543 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:03.278500    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:05.344111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:05.344079    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:15:06.325378 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325335    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:15:06.325730 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325713    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="storage-initializer"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325732    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="storage-initializer"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325753    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kube-rbac-proxy"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325776    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kube-rbac-proxy"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325802    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325811    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container"
Jul 10 19:15:06.325824 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325823    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="storage-initializer"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325831    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="storage-initializer"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325842    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325851    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325860    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kube-rbac-proxy"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325868    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kube-rbac-proxy"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325955    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kube-rbac-proxy"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325967    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kube-rbac-proxy"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325980    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae4fa15-4c11-4944-a7c0-f17bac5effaa" containerName="kserve-container"
Jul 10 19:15:06.326131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.325989    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7201858-6980-453d-8745-c8ff65e9a191" containerName="kserve-container"
Jul 10 19:15:06.329221 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.329200    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.331604 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.331585    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-358e7-predictor-serving-cert\""
Jul 10 19:15:06.331934 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.331913    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-358e7-kube-rbac-proxy-sar-config\""
Jul 10 19:15:06.332022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.331917    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:15:06.332022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.331916    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:15:06.338998 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.338975    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:15:06.437305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.437261    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.437689 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.437333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.437689 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.437381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4b8\" (UniqueName: \"kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.437689 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.437410    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.437689 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.437434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538296 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538262    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4b8\" (UniqueName: \"kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538309    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538496 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.538882 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.538858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.539174 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.539154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.539174 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.539165    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.540715 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.540697    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.546964 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.546941    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4b8\" (UniqueName: \"kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8\") pod \"isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") " pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.640594 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.640557    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:06.774479 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:06.774425    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:15:06.777335 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:15:06.777302    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9d6054_06e5_4f7f_abfe_c30d335fae19.slice/crio-6867dcc9b7a5330d8048c889b1710ab64103dd786cac220ea67e6ec5dd75e0c7 WatchSource:0}: Error finding container 6867dcc9b7a5330d8048c889b1710ab64103dd786cac220ea67e6ec5dd75e0c7: Status 404 returned error can't find the container with id 6867dcc9b7a5330d8048c889b1710ab64103dd786cac220ea67e6ec5dd75e0c7
Jul 10 19:15:07.373883 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:07.373846    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerStarted","Data":"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"}
Jul 10 19:15:07.374024 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:07.373915    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerStarted","Data":"6867dcc9b7a5330d8048c889b1710ab64103dd786cac220ea67e6ec5dd75e0c7"}
Jul 10 19:15:08.278608 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:08.278569    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:08.279109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:08.278689    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:15:11.399004 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:11.398967    2570 generic.go:356] "Generic (PLEG): container finished" podID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerID="796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d" exitCode=0
Jul 10 19:15:11.399376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:11.399044    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerDied","Data":"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"}
Jul 10 19:15:12.406075 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406041    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerStarted","Data":"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"}
Jul 10 19:15:12.406487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406112    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerStarted","Data":"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"}
Jul 10 19:15:12.406487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerStarted","Data":"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"}
Jul 10 19:15:12.406487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406423    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:12.406487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406483    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:12.406636 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.406518    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:12.408093 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.408061    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:12.408779 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.408746    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:12.430517 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:12.430469    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podStartSLOduration=2.773298662 podStartE2EDuration="6.430452802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65715414s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:15:12.428502681 +0000 UTC m=+996.562433347" watchObservedRunningTime="2026-07-10 19:15:12.430452802 +0000 UTC m=+996.564383469"
Jul 10 19:15:13.278527 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:13.278485    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:13.412492 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:13.412452    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:13.412946 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:13.412922    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:18.278066 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:18.278025    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:18.416438 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:18.416411    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:15:18.417030 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:18.416993    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:18.417465 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:18.417441    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:23.277796 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:23.277736    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:26.196957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.196931    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:15:26.303667 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.303625    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle\") pod \"9805083d-e472-4d19-a017-3bcc58721940\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") "
Jul 10 19:15:26.303859 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.303678    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") pod \"9805083d-e472-4d19-a017-3bcc58721940\" (UID: \"9805083d-e472-4d19-a017-3bcc58721940\") "
Jul 10 19:15:26.304046 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.304021    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9805083d-e472-4d19-a017-3bcc58721940" (UID: "9805083d-e472-4d19-a017-3bcc58721940"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:26.305754 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.305711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9805083d-e472-4d19-a017-3bcc58721940" (UID: "9805083d-e472-4d19-a017-3bcc58721940"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:26.405065 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.405024    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805083d-e472-4d19-a017-3bcc58721940-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:26.405065 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.405060    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9805083d-e472-4d19-a017-3bcc58721940-openshift-service-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:15:26.467660 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.467620    2570 generic.go:356] "Generic (PLEG): container finished" podID="9805083d-e472-4d19-a017-3bcc58721940" containerID="ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0" exitCode=0
Jul 10 19:15:26.467660 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.467663    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"
Jul 10 19:15:26.467960 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.467686    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" event={"ID":"9805083d-e472-4d19-a017-3bcc58721940","Type":"ContainerDied","Data":"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"}
Jul 10 19:15:26.467960 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.467736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk" event={"ID":"9805083d-e472-4d19-a017-3bcc58721940","Type":"ContainerDied","Data":"5fe2ff7d81f57a9de2311a3429637c4fd0ce684c2c5a82e47f9a9cbfa0e7de4f"}
Jul 10 19:15:26.467960 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.467782    2570 scope.go:117] "RemoveContainer" containerID="ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"
Jul 10 19:15:26.475853 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.475837    2570 scope.go:117] "RemoveContainer" containerID="ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"
Jul 10 19:15:26.476093 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:15:26.476072    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0\": container with ID starting with ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0 not found: ID does not exist" containerID="ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"
Jul 10 19:15:26.476151 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.476101    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0"} err="failed to get container status \"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0\": rpc error: code = NotFound desc = could not find container \"ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0\": container with ID starting with ed9f02e4658e1bcca0f1a69c7a797a66f941f6026506f68b564eead37d547ba0 not found: ID does not exist"
Jul 10 19:15:26.484084 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.484059    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:15:26.489614 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:26.489592    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-fed9d-65f5bbb578-5jmqk"]
Jul 10 19:15:28.414934 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:28.414895    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9805083d-e472-4d19-a017-3bcc58721940" path="/var/lib/kubelet/pods/9805083d-e472-4d19-a017-3bcc58721940/volumes"
Jul 10 19:15:28.417533 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:28.417503    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:28.418012 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:28.417992    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:38.417050 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:38.417007    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:38.417526 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:38.417498    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:48.417119 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:48.417075    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:48.417613 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:48.417569    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:15:58.417047 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:58.416997    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:15:58.417504 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:15:58.417423    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:16:08.417120 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:08.417081    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:16:08.417579 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:08.417485    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:16:18.417613 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:18.417568    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:16:18.418073 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:18.418045    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:16:28.417533 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:28.417457    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:16:28.417962 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:28.417738    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:16:41.346246 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.346210    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-358e7-predictor-9f5c55544-9gxsq_deb26912-ed75-416d-814c-1de8d6d5f0c1/kserve-container/0.log"
Jul 10 19:16:41.570200 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.570165    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:16:41.570605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.570439    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" containerID="cri-o://9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08" gracePeriod=30
Jul 10 19:16:41.570605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.570473    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" containerID="cri-o://0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370" gracePeriod=30
Jul 10 19:16:41.570605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.570492    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" containerID="cri-o://1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee" gracePeriod=30
Jul 10 19:16:41.658402 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.658368    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:16:41.658709 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.658698    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d"
Jul 10 19:16:41.658783 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.658711    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d"
Jul 10 19:16:41.658783 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.658782    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9805083d-e472-4d19-a017-3bcc58721940" containerName="model-chainer-raw-hpa-fed9d"
Jul 10 19:16:41.662170 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.662153    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.665480 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.665447    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dc419-kube-rbac-proxy-sar-config\""
Jul 10 19:16:41.665725 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.665708    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dc419-predictor-serving-cert\""
Jul 10 19:16:41.676687 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.676650    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:16:41.713192 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.713154    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:16:41.713393 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.713368    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kserve-container" containerID="cri-o://0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5" gracePeriod=30
Jul 10 19:16:41.713457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.713431    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kube-rbac-proxy" containerID="cri-o://47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b" gracePeriod=30
Jul 10 19:16:41.735988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.735959    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.736123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.736002    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.736123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.736041    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.736123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.736069    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.736123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.736103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm42z\" (UniqueName: \"kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.778455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.778423    2570 generic.go:356] "Generic (PLEG): container finished" podID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerID="1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee" exitCode=2
Jul 10 19:16:41.778613 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.778497    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerDied","Data":"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"}
Jul 10 19:16:41.836957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.836928    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837105 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.836968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837105 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm42z\" (UniqueName: \"kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837105 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837254 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837211    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837402 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837376    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837686 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.837826 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.837807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.839414 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.839391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.845377 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.845352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm42z\" (UniqueName: \"kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z\") pod \"isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:41.961671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.961643    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:16:41.978716 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:41.978679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:42.038622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.038588    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls\") pod \"deb26912-ed75-416d-814c-1de8d6d5f0c1\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") "
Jul 10 19:16:42.038754 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.038683    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clmhb\" (UniqueName: \"kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb\") pod \"deb26912-ed75-416d-814c-1de8d6d5f0c1\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") "
Jul 10 19:16:42.038754 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.038721    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config\") pod \"deb26912-ed75-416d-814c-1de8d6d5f0c1\" (UID: \"deb26912-ed75-416d-814c-1de8d6d5f0c1\") "
Jul 10 19:16:42.039283 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.039253    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "deb26912-ed75-416d-814c-1de8d6d5f0c1" (UID: "deb26912-ed75-416d-814c-1de8d6d5f0c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:16:42.041512 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.041472    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "deb26912-ed75-416d-814c-1de8d6d5f0c1" (UID: "deb26912-ed75-416d-814c-1de8d6d5f0c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:16:42.041623 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.041483    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb" (OuterVolumeSpecName: "kube-api-access-clmhb") pod "deb26912-ed75-416d-814c-1de8d6d5f0c1" (UID: "deb26912-ed75-416d-814c-1de8d6d5f0c1"). InnerVolumeSpecName "kube-api-access-clmhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:16:42.112196 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.112162    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:16:42.114594 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:16:42.114570    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd52b2b9_efec_49df_8452_b38247a60cc2.slice/crio-ada7194790c8c9d1a298f17d6b28fc76e3798dc974a4b76c9a13701c48d6c4f9 WatchSource:0}: Error finding container ada7194790c8c9d1a298f17d6b28fc76e3798dc974a4b76c9a13701c48d6c4f9: Status 404 returned error can't find the container with id ada7194790c8c9d1a298f17d6b28fc76e3798dc974a4b76c9a13701c48d6c4f9
Jul 10 19:16:42.140213 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.140178    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmhb\" (UniqueName: \"kubernetes.io/projected/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-api-access-clmhb\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:16:42.140305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.140220    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deb26912-ed75-416d-814c-1de8d6d5f0c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:16:42.140305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.140236    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deb26912-ed75-416d-814c-1de8d6d5f0c1-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:16:42.784040 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.783942    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerStarted","Data":"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"}
Jul 10 19:16:42.784040 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.784008    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerStarted","Data":"ada7194790c8c9d1a298f17d6b28fc76e3798dc974a4b76c9a13701c48d6c4f9"}
Jul 10 19:16:42.785436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785408    2570 generic.go:356] "Generic (PLEG): container finished" podID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerID="47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b" exitCode=2
Jul 10 19:16:42.785436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785431    2570 generic.go:356] "Generic (PLEG): container finished" podID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerID="0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5" exitCode=2
Jul 10 19:16:42.785579 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785446    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"
Jul 10 19:16:42.785579 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785491    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerDied","Data":"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"}
Jul 10 19:16:42.785579 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785538    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerDied","Data":"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"}
Jul 10 19:16:42.785579 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq" event={"ID":"deb26912-ed75-416d-814c-1de8d6d5f0c1","Type":"ContainerDied","Data":"4c180a061bd38eb1c33920f589f5748c46692decf884cfbfd05c054bf925e793"}
Jul 10 19:16:42.785730 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.785582    2570 scope.go:117] "RemoveContainer" containerID="47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"
Jul 10 19:16:42.793708 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.793687    2570 scope.go:117] "RemoveContainer" containerID="0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"
Jul 10 19:16:42.800973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.800958    2570 scope.go:117] "RemoveContainer" containerID="47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"
Jul 10 19:16:42.801219 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:16:42.801203    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b\": container with ID starting with 47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b not found: ID does not exist" containerID="47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"
Jul 10 19:16:42.801261 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801230    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"} err="failed to get container status \"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b\": rpc error: code = NotFound desc = could not find container \"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b\": container with ID starting with 47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b not found: ID does not exist"
Jul 10 19:16:42.801261 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801249    2570 scope.go:117] "RemoveContainer" containerID="0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"
Jul 10 19:16:42.801499 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:16:42.801477    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5\": container with ID starting with 0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5 not found: ID does not exist" containerID="0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"
Jul 10 19:16:42.801553 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801508    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"} err="failed to get container status \"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5\": rpc error: code = NotFound desc = could not find container \"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5\": container with ID starting with 0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5 not found: ID does not exist"
Jul 10 19:16:42.801553 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801525    2570 scope.go:117] "RemoveContainer" containerID="47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"
Jul 10 19:16:42.801717 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801701    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b"} err="failed to get container status \"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b\": rpc error: code = NotFound desc = could not find container \"47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b\": container with ID starting with 47796171d27e076c0fecc40792882c850ac5bec39107e1eb4a9c03d329b7e02b not found: ID does not exist"
Jul 10 19:16:42.801774 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801719    2570 scope.go:117] "RemoveContainer" containerID="0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"
Jul 10 19:16:42.801917 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.801900    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5"} err="failed to get container status \"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5\": rpc error: code = NotFound desc = could not find container \"0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5\": container with ID starting with 0e41fd0f059f034c4f8db20d861a0d56158fe6ff585d45f69c34cfc91e644ff5 not found: ID does not exist"
Jul 10 19:16:42.815507 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.815471    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:16:42.821555 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:42.821527    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-358e7-predictor-9f5c55544-9gxsq"]
Jul 10 19:16:43.413346 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:43.413298    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:16:44.414931 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:44.414897    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" path="/var/lib/kubelet/pods/deb26912-ed75-416d-814c-1de8d6d5f0c1/volumes"
Jul 10 19:16:46.805022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:46.804985    2570 generic.go:356] "Generic (PLEG): container finished" podID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerID="745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f" exitCode=0
Jul 10 19:16:46.805465 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:46.805058    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerDied","Data":"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"}
Jul 10 19:16:46.807477 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:46.807453    2570 generic.go:356] "Generic (PLEG): container finished" podID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerID="9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08" exitCode=0
Jul 10 19:16:46.807580 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:46.807488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerDied","Data":"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"}
Jul 10 19:16:47.813423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.813387    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerStarted","Data":"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"}
Jul 10 19:16:47.813830 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.813452    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerStarted","Data":"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"}
Jul 10 19:16:47.813830 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.813710    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:47.813898 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.813863    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:47.815100 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.815045    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:16:47.860557 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:47.860503    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podStartSLOduration=3.193661655 podStartE2EDuration="6.860486432s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.666824777s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:16:47.857155221 +0000 UTC m=+1091.991085889" watchObservedRunningTime="2026-07-10 19:16:47.860486432 +0000 UTC m=+1091.994417098"
Jul 10 19:16:48.412948 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:48.412886    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:16:48.416991 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:48.416967    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:16:48.417350 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:48.417331    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:16:48.817410 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:48.817317    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:16:53.413578 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:53.413527    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:16:53.413965 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:53.413709    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:16:53.822649 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:53.822571    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:16:53.823109 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:53.823071    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:16:58.412672 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:58.412635    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:16:58.417817 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:58.417791    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:16:58.418145 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:16:58.418121    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:17:03.412693 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:03.412648    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:17:03.823294 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:03.823205    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:08.412781 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:08.412721    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 10 19:17:08.417864 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:08.417824    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 10 19:17:08.418023 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:08.417937    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:17:08.418152 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:08.418132    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 19:17:08.418229 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:08.418215    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:17:11.717497 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.717466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:17:11.781915 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.781881    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location\") pod \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") "
Jul 10 19:17:11.781915 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.781920    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls\") pod \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") "
Jul 10 19:17:11.782164 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.781955    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf4b8\" (UniqueName: \"kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8\") pod \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") "
Jul 10 19:17:11.782164 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.781977    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config\") pod \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") "
Jul 10 19:17:11.782164 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.782037    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert\") pod \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\" (UID: \"2d9d6054-06e5-4f7f-abfe-c30d335fae19\") "
Jul 10 19:17:11.782375 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.782338    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d9d6054-06e5-4f7f-abfe-c30d335fae19" (UID: "2d9d6054-06e5-4f7f-abfe-c30d335fae19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:11.782487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.782361    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d9d6054-06e5-4f7f-abfe-c30d335fae19" (UID: "2d9d6054-06e5-4f7f-abfe-c30d335fae19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:17:11.782487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.782428    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d9d6054-06e5-4f7f-abfe-c30d335fae19" (UID: "2d9d6054-06e5-4f7f-abfe-c30d335fae19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:17:11.784054 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.784036    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d9d6054-06e5-4f7f-abfe-c30d335fae19" (UID: "2d9d6054-06e5-4f7f-abfe-c30d335fae19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:17:11.784183 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.784163    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8" (OuterVolumeSpecName: "kube-api-access-cf4b8") pod "2d9d6054-06e5-4f7f-abfe-c30d335fae19" (UID: "2d9d6054-06e5-4f7f-abfe-c30d335fae19"). InnerVolumeSpecName "kube-api-access-cf4b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:17:11.883244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.883142    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:17:11.883244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.883189    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:17:11.883244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.883200    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9d6054-06e5-4f7f-abfe-c30d335fae19-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:17:11.883244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.883210    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf4b8\" (UniqueName: \"kubernetes.io/projected/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-api-access-cf4b8\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:17:11.883244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.883218    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9d6054-06e5-4f7f-abfe-c30d335fae19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:17:11.905592 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.905557    2570 generic.go:356] "Generic (PLEG): container finished" podID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerID="0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370" exitCode=0
Jul 10 19:17:11.905746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.905598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerDied","Data":"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"}
Jul 10 19:17:11.905746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.905618    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"
Jul 10 19:17:11.905746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.905663    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn" event={"ID":"2d9d6054-06e5-4f7f-abfe-c30d335fae19","Type":"ContainerDied","Data":"6867dcc9b7a5330d8048c889b1710ab64103dd786cac220ea67e6ec5dd75e0c7"}
Jul 10 19:17:11.905746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.905696    2570 scope.go:117] "RemoveContainer" containerID="0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"
Jul 10 19:17:11.914602 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.914583    2570 scope.go:117] "RemoveContainer" containerID="1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"
Jul 10 19:17:11.928161 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.928136    2570 scope.go:117] "RemoveContainer" containerID="9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"
Jul 10 19:17:11.931747 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.931718    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:17:11.936597 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.936570    2570 scope.go:117] "RemoveContainer" containerID="796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"
Jul 10 19:17:11.936684 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.936591    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-358e7-predictor-74b9cfb87c-sn8qn"]
Jul 10 19:17:11.943884 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.943847    2570 scope.go:117] "RemoveContainer" containerID="0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"
Jul 10 19:17:11.944249 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:17:11.944230    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370\": container with ID starting with 0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370 not found: ID does not exist" containerID="0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"
Jul 10 19:17:11.944364 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944257    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370"} err="failed to get container status \"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370\": rpc error: code = NotFound desc = could not find container \"0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370\": container with ID starting with 0bb4c5972da499c400de6136a639741a9fa47b1fd984085dbaabc69838edb370 not found: ID does not exist"
Jul 10 19:17:11.944364 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944279    2570 scope.go:117] "RemoveContainer" containerID="1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"
Jul 10 19:17:11.944528 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:17:11.944511    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee\": container with ID starting with 1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee not found: ID does not exist" containerID="1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"
Jul 10 19:17:11.944560 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944535    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee"} err="failed to get container status \"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee\": rpc error: code = NotFound desc = could not find container \"1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee\": container with ID starting with 1c1bee44c35e01290a7bedebac04e25a56cdd2bb955470ac4dbb6b1da62255ee not found: ID does not exist"
Jul 10 19:17:11.944560 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944551    2570 scope.go:117] "RemoveContainer" containerID="9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"
Jul 10 19:17:11.944816 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:17:11.944800    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08\": container with ID starting with 9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08 not found: ID does not exist" containerID="9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"
Jul 10 19:17:11.944857 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944820    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08"} err="failed to get container status \"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08\": rpc error: code = NotFound desc = could not find container \"9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08\": container with ID starting with 9f4c05b89dbc0d5cb255e91a715fae59e898613648b00ed0cbf92d8fa1b27a08 not found: ID does not exist"
Jul 10 19:17:11.944857 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.944833    2570 scope.go:117] "RemoveContainer" containerID="796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"
Jul 10 19:17:11.945044 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:17:11.945027    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d\": container with ID starting with 796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d not found: ID does not exist" containerID="796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"
Jul 10 19:17:11.945082 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:11.945049    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d"} err="failed to get container status \"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d\": rpc error: code = NotFound desc = could not find container \"796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d\": container with ID starting with 796a8ae782d30b30c0dc715dca5c65b61ffc4c0730fc75d204253d5b732d499d not found: ID does not exist"
Jul 10 19:17:12.414140 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:12.414107    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" path="/var/lib/kubelet/pods/2d9d6054-06e5-4f7f-abfe-c30d335fae19/volumes"
Jul 10 19:17:13.823428 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:13.823384    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:23.823611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:23.823568    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:33.823332 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:33.823293    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:43.823011 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:43.822968    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:53.823428 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:53.823385    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:17:55.409793 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:17:55.409724    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:05.409675 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:05.409637    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:15.409915 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:15.409866    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:25.410696 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:25.410649    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:35.409695 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:35.409649    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:36.360668 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:36.360632    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:18:36.362520 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:36.362494    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:18:45.409995 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:45.409945    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:18:55.409896 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:18:55.409848    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:19:05.410532 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:05.410487    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:19:15.410503 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:15.410455    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:19:19.410634 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:19.410604    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:19:21.834644 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:21.834607    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:19:21.835543 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:21.835477    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" containerID="cri-o://c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9" gracePeriod=30
Jul 10 19:19:21.835942 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:21.835915    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy" containerID="cri-o://ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805" gracePeriod=30
Jul 10 19:19:22.412267 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:22.412220    2570 generic.go:356] "Generic (PLEG): container finished" podID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerID="ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805" exitCode=2
Jul 10 19:19:22.413735 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:22.413709    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerDied","Data":"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"}
Jul 10 19:19:23.817850 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:23.817806    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 10 19:19:27.003661 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.003619    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.003961    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kserve-container"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.003973    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kserve-container"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.003985    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.003994    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004008    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004014    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004023    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004027    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004035    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kube-rbac-proxy"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004039    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kube-rbac-proxy"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004048    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="storage-initializer"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004054    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="storage-initializer"
Jul 10 19:19:27.004111 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004112    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kserve-container"
Jul 10 19:19:27.004493 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004122    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kserve-container"
Jul 10 19:19:27.004493 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004129    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="kube-rbac-proxy"
Jul 10 19:19:27.004493 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004135    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9d6054-06e5-4f7f-abfe-c30d335fae19" containerName="agent"
Jul 10 19:19:27.004493 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.004142    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb26912-ed75-416d-814c-1de8d6d5f0c1" containerName="kube-rbac-proxy"
Jul 10 19:19:27.007352 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.007335    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.009986 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.009964    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-147ca1-predictor-serving-cert\""
Jul 10 19:19:27.010127 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.010032    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-147ca1-kube-rbac-proxy-sar-config\""
Jul 10 19:19:27.017366 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.017339    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:19:27.075696 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.075660    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.075913 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.075712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.075913 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.075758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.075913 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.075804    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.075913 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.075841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xlp\" (UniqueName: \"kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176443 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176406    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176462    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xlp\" (UniqueName: \"kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176517    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176558    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176622 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.176999 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.176974    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.177169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.177154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.177222 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.177179    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.179094 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.179073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.184549 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.184515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xlp\" (UniqueName: \"kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp\") pod \"isvc-primary-147ca1-predictor-86b99c977c-956fs\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") " pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.317590 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.317489    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:27.448443 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:27.448297    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:19:27.451097 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:19:27.451069    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340d0984_a7ef_4996_99cb_cc7bf0aa4af9.slice/crio-fd38956cd6dfb954ee2cc74b00944c1f5b081a65cb0a417f2aa330a773e99797 WatchSource:0}: Error finding container fd38956cd6dfb954ee2cc74b00944c1f5b081a65cb0a417f2aa330a773e99797: Status 404 returned error can't find the container with id fd38956cd6dfb954ee2cc74b00944c1f5b081a65cb0a417f2aa330a773e99797
Jul 10 19:19:28.441576 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:28.441353    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerStarted","Data":"7b7e89602e177dec38e76d82c1c8cb93e99b5d9fc416fab110952e1cbc4325b7"}
Jul 10 19:19:28.441576 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:28.441476    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerStarted","Data":"fd38956cd6dfb954ee2cc74b00944c1f5b081a65cb0a417f2aa330a773e99797"}
Jul 10 19:19:28.818080 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:28.817985    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 10 19:19:29.410814 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:29.410746    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 10 19:19:31.454076 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.454039    2570 generic.go:356] "Generic (PLEG): container finished" podID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerID="7b7e89602e177dec38e76d82c1c8cb93e99b5d9fc416fab110952e1cbc4325b7" exitCode=0
Jul 10 19:19:31.454467 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.454109    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerDied","Data":"7b7e89602e177dec38e76d82c1c8cb93e99b5d9fc416fab110952e1cbc4325b7"}
Jul 10 19:19:31.682358 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.682333    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:19:31.711947 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.711880    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config\") pod \"dd52b2b9-efec-49df-8452-b38247a60cc2\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") "
Jul 10 19:19:31.712094 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.711946    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert\") pod \"dd52b2b9-efec-49df-8452-b38247a60cc2\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") "
Jul 10 19:19:31.712094 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712006    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls\") pod \"dd52b2b9-efec-49df-8452-b38247a60cc2\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") "
Jul 10 19:19:31.712094 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm42z\" (UniqueName: \"kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z\") pod \"dd52b2b9-efec-49df-8452-b38247a60cc2\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") "
Jul 10 19:19:31.712094 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location\") pod \"dd52b2b9-efec-49df-8452-b38247a60cc2\" (UID: \"dd52b2b9-efec-49df-8452-b38247a60cc2\") "
Jul 10 19:19:31.712309 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712257    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd52b2b9-efec-49df-8452-b38247a60cc2" (UID: "dd52b2b9-efec-49df-8452-b38247a60cc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:19:31.712462 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712364    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd52b2b9-efec-49df-8452-b38247a60cc2" (UID: "dd52b2b9-efec-49df-8452-b38247a60cc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:19:31.712517 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.712501    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd52b2b9-efec-49df-8452-b38247a60cc2" (UID: "dd52b2b9-efec-49df-8452-b38247a60cc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:31.714145 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.714118    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z" (OuterVolumeSpecName: "kube-api-access-mm42z") pod "dd52b2b9-efec-49df-8452-b38247a60cc2" (UID: "dd52b2b9-efec-49df-8452-b38247a60cc2"). InnerVolumeSpecName "kube-api-access-mm42z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:19:31.714244 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.714149    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd52b2b9-efec-49df-8452-b38247a60cc2" (UID: "dd52b2b9-efec-49df-8452-b38247a60cc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:19:31.813343 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.813311    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd52b2b9-efec-49df-8452-b38247a60cc2-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:19:31.813343 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.813337    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:19:31.813343 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.813348    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd52b2b9-efec-49df-8452-b38247a60cc2-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:19:31.813562 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.813357    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd52b2b9-efec-49df-8452-b38247a60cc2-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:19:31.813562 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:31.813366    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm42z\" (UniqueName: \"kubernetes.io/projected/dd52b2b9-efec-49df-8452-b38247a60cc2-kube-api-access-mm42z\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:19:32.460927 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.460887    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerStarted","Data":"57aa8ecad1244459784ede3205cc1cfb9752b2061e1e1c5bdd26c612499b7d09"}
Jul 10 19:19:32.461337 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.460963    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerStarted","Data":"1e778cb84189c71398799dcabb79799fc51149dbc6725670ab436ec414c5b27d"}
Jul 10 19:19:32.461337 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.461263    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:32.461557 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.461533    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:32.462844 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462819    2570 generic.go:356] "Generic (PLEG): container finished" podID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerID="c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9" exitCode=0
Jul 10 19:19:32.462957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462859    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:19:32.462957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462875    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerDied","Data":"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"}
Jul 10 19:19:32.462957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462884    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"
Jul 10 19:19:32.462957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462922    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm" event={"ID":"dd52b2b9-efec-49df-8452-b38247a60cc2","Type":"ContainerDied","Data":"ada7194790c8c9d1a298f17d6b28fc76e3798dc974a4b76c9a13701c48d6c4f9"}
Jul 10 19:19:32.463132 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.462960    2570 scope.go:117] "RemoveContainer" containerID="ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"
Jul 10 19:19:32.470831 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.470814    2570 scope.go:117] "RemoveContainer" containerID="c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"
Jul 10 19:19:32.477880 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.477860    2570 scope.go:117] "RemoveContainer" containerID="745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"
Jul 10 19:19:32.484234 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.484173    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podStartSLOduration=2.951869059 podStartE2EDuration="6.484154338s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.532285279s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:32.483093776 +0000 UTC m=+1256.617024443" watchObservedRunningTime="2026-07-10 19:19:32.484154338 +0000 UTC m=+1256.618085006"
Jul 10 19:19:32.486038 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486021    2570 scope.go:117] "RemoveContainer" containerID="ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"
Jul 10 19:19:32.486369 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:19:32.486309    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805\": container with ID starting with ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805 not found: ID does not exist" containerID="ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"
Jul 10 19:19:32.486456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486384    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805"} err="failed to get container status \"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805\": rpc error: code = NotFound desc = could not find container \"ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805\": container with ID starting with ab0d8a1742ba45ba1004c91ae2f2598692ca49c1f340a744989a52cecd234805 not found: ID does not exist"
Jul 10 19:19:32.486456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486403    2570 scope.go:117] "RemoveContainer" containerID="c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"
Jul 10 19:19:32.486676 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:19:32.486660    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9\": container with ID starting with c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9 not found: ID does not exist" containerID="c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"
Jul 10 19:19:32.486716 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486681    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9"} err="failed to get container status \"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9\": rpc error: code = NotFound desc = could not find container \"c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9\": container with ID starting with c5766a76b10a5957c4d17167fb1215f38ef1cf13af628b30376fc9e7886b4dd9 not found: ID does not exist"
Jul 10 19:19:32.486716 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486703    2570 scope.go:117] "RemoveContainer" containerID="745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"
Jul 10 19:19:32.486966 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:19:32.486947    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f\": container with ID starting with 745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f not found: ID does not exist" containerID="745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"
Jul 10 19:19:32.487023 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.486967    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f"} err="failed to get container status \"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f\": rpc error: code = NotFound desc = could not find container \"745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f\": container with ID starting with 745b77394dff70ac55d2968ef66d6d8e4e19b4877ae732e224d6a1da9397f70f not found: ID does not exist"
Jul 10 19:19:32.497821 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.497788    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:19:32.505234 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:32.505202    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc419-predictor-7ddf54994b-8r5bm"]
Jul 10 19:19:33.468320 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:33.468283    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:19:34.413880 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:34.413849    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" path="/var/lib/kubelet/pods/dd52b2b9-efec-49df-8452-b38247a60cc2/volumes"
Jul 10 19:19:34.471973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:34.471933    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:19:39.476399 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:39.476365    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:19:39.476926 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:39.476894    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:19:49.477025 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:49.476982    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:19:59.477611 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:19:59.477568    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:20:09.477251 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:09.477209    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:20:19.477018 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:19.476980    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:20:29.476847 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:29.476807    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:20:39.476811 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:39.476752    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:20:49.478377 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:49.478344    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:20:57.180746 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.180710    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:20:57.181223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181179    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy"
Jul 10 19:20:57.181223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181197    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy"
Jul 10 19:20:57.181223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181220    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="storage-initializer"
Jul 10 19:20:57.181374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181228    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="storage-initializer"
Jul 10 19:20:57.181374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181243    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container"
Jul 10 19:20:57.181374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181252    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container"
Jul 10 19:20:57.181374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181341    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kube-rbac-proxy"
Jul 10 19:20:57.181374 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.181358    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd52b2b9-efec-49df-8452-b38247a60cc2" containerName="kserve-container"
Jul 10 19:20:57.184395 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.184374    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.186862 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.186841    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-147ca1-kube-rbac-proxy-sar-config\""
Jul 10 19:20:57.186989 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.186938    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-147ca1-predictor-serving-cert\""
Jul 10 19:20:57.187052 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.187036    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-147ca1-dockercfg-nlzwz\""
Jul 10 19:20:57.187174 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.187113    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-147ca1\""
Jul 10 19:20:57.194306 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.194286    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:20:57.305644 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.305606    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.305849 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.305668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.305849 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.305761    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.305849 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.305828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.305968 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.305850    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7bp\" (UniqueName: \"kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407136 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407494 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7bp\" (UniqueName: \"kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407786 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407728    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407926 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407904    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.407979 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.407918    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.409783 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.409749    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.416288 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.416268    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7bp\" (UniqueName: \"kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp\") pod \"isvc-secondary-147ca1-predictor-5d9994b64b-28qqq\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") " pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.495335 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.495240    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:20:57.623057 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.622886    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:20:57.625642 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:20:57.625613    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4ff367_de06_45b5_88e0_3ae4b0ef27e8.slice/crio-9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee WatchSource:0}: Error finding container 9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee: Status 404 returned error can't find the container with id 9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee
Jul 10 19:20:57.790475 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.790393    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerStarted","Data":"ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"}
Jul 10 19:20:57.790475 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:20:57.790448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerStarted","Data":"9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee"}
Jul 10 19:21:03.814784 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:03.814745    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/0.log"
Jul 10 19:21:03.815160 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:03.814801    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerID="ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9" exitCode=1
Jul 10 19:21:03.815160 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:03.814872    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerDied","Data":"ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"}
Jul 10 19:21:04.820516 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:04.820489    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/0.log"
Jul 10 19:21:04.820922 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:04.820584    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerStarted","Data":"6ef33dddb37b42a5a2b2287e511317c3e97aef2d4dc1fdef8d462d3f5d98548c"}
Jul 10 19:21:09.840115 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.840086    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/1.log"
Jul 10 19:21:09.840566 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.840484    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/0.log"
Jul 10 19:21:09.840566 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.840527    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerID="6ef33dddb37b42a5a2b2287e511317c3e97aef2d4dc1fdef8d462d3f5d98548c" exitCode=1
Jul 10 19:21:09.840566 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.840560    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerDied","Data":"6ef33dddb37b42a5a2b2287e511317c3e97aef2d4dc1fdef8d462d3f5d98548c"}
Jul 10 19:21:09.840721 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.840633    2570 scope.go:117] "RemoveContainer" containerID="ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"
Jul 10 19:21:09.841051 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.841029    2570 scope.go:117] "RemoveContainer" containerID="ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"
Jul 10 19:21:09.851244 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:21:09.851222    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_kserve-ci-e2e-test(ea4ff367-de06-45b5-88e0-3ae4b0ef27e8)\"" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"
Jul 10 19:21:09.851322 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:21:09.851297    2570 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_kserve-ci-e2e-test_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8_0 in pod sandbox 9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee from index: no such id: 'ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9'" containerID="ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"
Jul 10 19:21:09.851362 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:09.851334    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_kserve-ci-e2e-test_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8_0 in pod sandbox 9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee from index: no such id: 'ed338515b711df7e9f6b64b6340228463557591c4c0e0cfc6fdb0815eb94edb9'"
Jul 10 19:21:10.846380 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:10.846350    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/1.log"
Jul 10 19:21:15.226523 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.226491    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:21:15.281522 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.281421    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:21:15.281777 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.281734    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" containerID="cri-o://1e778cb84189c71398799dcabb79799fc51149dbc6725670ab436ec414c5b27d" gracePeriod=30
Jul 10 19:21:15.282068 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.281783    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kube-rbac-proxy" containerID="cri-o://57aa8ecad1244459784ede3205cc1cfb9752b2061e1e1c5bdd26c612499b7d09" gracePeriod=30
Jul 10 19:21:15.401038 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.401018    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/1.log"
Jul 10 19:21:15.403169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.403152    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:21:15.564671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.564580    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls\") pod \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") "
Jul 10 19:21:15.564671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.564639    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config\") pod \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") "
Jul 10 19:21:15.564671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.564673    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7bp\" (UniqueName: \"kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp\") pod \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") "
Jul 10 19:21:15.564973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.564848    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location\") pod \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") "
Jul 10 19:21:15.564973 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.564949    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert\") pod \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\" (UID: \"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8\") "
Jul 10 19:21:15.565104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.565082    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" (UID: "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:15.565104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.565091    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" (UID: "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:15.565239 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.565216    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.565239 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.565239    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.565370 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.565341    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" (UID: "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:15.566911 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.566882    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" (UID: "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:15.566911 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.566889    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp" (OuterVolumeSpecName: "kube-api-access-nw7bp") pod "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" (UID: "ea4ff367-de06-45b5-88e0-3ae4b0ef27e8"). InnerVolumeSpecName "kube-api-access-nw7bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:15.666034 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.665996    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.666034 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.666026    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7bp\" (UniqueName: \"kubernetes.io/projected/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-kube-api-access-nw7bp\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.666034 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.666036    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:15.872425 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.872343    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-147ca1-predictor-5d9994b64b-28qqq_ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/storage-initializer/1.log"
Jul 10 19:21:15.872591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.872449    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"
Jul 10 19:21:15.872591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.872463    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq" event={"ID":"ea4ff367-de06-45b5-88e0-3ae4b0ef27e8","Type":"ContainerDied","Data":"9cbc2c6616331bbb99c093e9b43b5270abf2d17cda44d923c55097d3898d4fee"}
Jul 10 19:21:15.872591 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.872531    2570 scope.go:117] "RemoveContainer" containerID="6ef33dddb37b42a5a2b2287e511317c3e97aef2d4dc1fdef8d462d3f5d98548c"
Jul 10 19:21:15.874916 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.874891    2570 generic.go:356] "Generic (PLEG): container finished" podID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerID="57aa8ecad1244459784ede3205cc1cfb9752b2061e1e1c5bdd26c612499b7d09" exitCode=2
Jul 10 19:21:15.875055 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.874974    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerDied","Data":"57aa8ecad1244459784ede3205cc1cfb9752b2061e1e1c5bdd26c612499b7d09"}
Jul 10 19:21:15.927139 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.927108    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:21:15.933569 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:15.933545    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-147ca1-predictor-5d9994b64b-28qqq"]
Jul 10 19:21:16.414527 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:16.414498    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" path="/var/lib/kubelet/pods/ea4ff367-de06-45b5-88e0-3ae4b0ef27e8/volumes"
Jul 10 19:21:19.472513 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:19.472475    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 10 19:21:19.476919 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:19.476890    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 10 19:21:19.894727 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:19.894694    2570 generic.go:356] "Generic (PLEG): container finished" podID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerID="1e778cb84189c71398799dcabb79799fc51149dbc6725670ab436ec414c5b27d" exitCode=0
Jul 10 19:21:19.894899 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:19.894752    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerDied","Data":"1e778cb84189c71398799dcabb79799fc51149dbc6725670ab436ec414c5b27d"}
Jul 10 19:21:19.931493 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:19.931466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:21:20.103794 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.103685    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls\") pod \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") "
Jul 10 19:21:20.103794 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.103729    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location\") pod \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") "
Jul 10 19:21:20.104028 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.103799    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9xlp\" (UniqueName: \"kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp\") pod \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") "
Jul 10 19:21:20.104028 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.103833    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert\") pod \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") "
Jul 10 19:21:20.104028 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.103872    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config\") pod \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\" (UID: \"340d0984-a7ef-4996-99cb-cc7bf0aa4af9\") "
Jul 10 19:21:20.104210 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.104153    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "340d0984-a7ef-4996-99cb-cc7bf0aa4af9" (UID: "340d0984-a7ef-4996-99cb-cc7bf0aa4af9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:20.104255 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.104202    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "340d0984-a7ef-4996-99cb-cc7bf0aa4af9" (UID: "340d0984-a7ef-4996-99cb-cc7bf0aa4af9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:20.104255 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.104232    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "340d0984-a7ef-4996-99cb-cc7bf0aa4af9" (UID: "340d0984-a7ef-4996-99cb-cc7bf0aa4af9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:20.105978 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.105954    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp" (OuterVolumeSpecName: "kube-api-access-t9xlp") pod "340d0984-a7ef-4996-99cb-cc7bf0aa4af9" (UID: "340d0984-a7ef-4996-99cb-cc7bf0aa4af9"). InnerVolumeSpecName "kube-api-access-t9xlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:20.106059 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.105953    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "340d0984-a7ef-4996-99cb-cc7bf0aa4af9" (UID: "340d0984-a7ef-4996-99cb-cc7bf0aa4af9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:20.205369 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.205331    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xlp\" (UniqueName: \"kubernetes.io/projected/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-api-access-t9xlp\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:20.205369 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.205363    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:20.205369 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.205372    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:20.205602 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.205381    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:20.205602 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.205390    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/340d0984-a7ef-4996-99cb-cc7bf0aa4af9-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:20.445988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.445957    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:20.446252 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446240    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kube-rbac-proxy"
Jul 10 19:21:20.446290 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446254    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kube-rbac-proxy"
Jul 10 19:21:20.446290 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446267    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container"
Jul 10 19:21:20.446290 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446272    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container"
Jul 10 19:21:20.446290 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446282    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="storage-initializer"
Jul 10 19:21:20.446290 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446288    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446296    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446301    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446313    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446319    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446363    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kube-rbac-proxy"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446369    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" containerName="kserve-container"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446375    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.446441 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.446381    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4ff367-de06-45b5-88e0-3ae4b0ef27e8" containerName="storage-initializer"
Jul 10 19:21:20.449124 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.449108    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.451681 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.451664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e3c30a\""
Jul 10 19:21:20.451823 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.451664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e3c30a-predictor-serving-cert\""
Jul 10 19:21:20.451988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.451967    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e3c30a-kube-rbac-proxy-sar-config\""
Jul 10 19:21:20.452213 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.452198    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e3c30a-dockercfg-5w8cg\""
Jul 10 19:21:20.461209 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.461189    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:20.608615 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.608580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.608615 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.608615    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.609104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.608647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.609104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.608777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.609104 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.608818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.709816 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.709717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.709816 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.709754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.709816 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.709803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.710095 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.709821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.710095 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.709844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.710282 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.710260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.710443 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.710425    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.710477 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.710445    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.712268 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.712249    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.719172 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.719146    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf\") pod \"isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.759269 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.759244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:20.886705 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.886676    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:20.888582 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:21:20.888553    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55984685_7f0b_473b_a35d_20ad505217a6.slice/crio-cde3b83caee1b6ec58b5e3e3c78b1875b032d69ee5ce9a5455d797224f831418 WatchSource:0}: Error finding container cde3b83caee1b6ec58b5e3e3c78b1875b032d69ee5ce9a5455d797224f831418: Status 404 returned error can't find the container with id cde3b83caee1b6ec58b5e3e3c78b1875b032d69ee5ce9a5455d797224f831418
Jul 10 19:21:20.900273 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.900251    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"
Jul 10 19:21:20.900381 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.900262    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs" event={"ID":"340d0984-a7ef-4996-99cb-cc7bf0aa4af9","Type":"ContainerDied","Data":"fd38956cd6dfb954ee2cc74b00944c1f5b081a65cb0a417f2aa330a773e99797"}
Jul 10 19:21:20.900381 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.900337    2570 scope.go:117] "RemoveContainer" containerID="57aa8ecad1244459784ede3205cc1cfb9752b2061e1e1c5bdd26c612499b7d09"
Jul 10 19:21:20.901515 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.901475    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerStarted","Data":"cde3b83caee1b6ec58b5e3e3c78b1875b032d69ee5ce9a5455d797224f831418"}
Jul 10 19:21:20.912634 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.912617    2570 scope.go:117] "RemoveContainer" containerID="1e778cb84189c71398799dcabb79799fc51149dbc6725670ab436ec414c5b27d"
Jul 10 19:21:20.922583 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.922557    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:21:20.923462 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.923447    2570 scope.go:117] "RemoveContainer" containerID="7b7e89602e177dec38e76d82c1c8cb93e99b5d9fc416fab110952e1cbc4325b7"
Jul 10 19:21:20.925877 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:20.925856    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-147ca1-predictor-86b99c977c-956fs"]
Jul 10 19:21:21.907756 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:21.907716    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerStarted","Data":"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"}
Jul 10 19:21:22.414152 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:22.414121    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340d0984-a7ef-4996-99cb-cc7bf0aa4af9" path="/var/lib/kubelet/pods/340d0984-a7ef-4996-99cb-cc7bf0aa4af9/volumes"
Jul 10 19:21:25.925972 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:25.925944    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/0.log"
Jul 10 19:21:25.926340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:25.925981    2570 generic.go:356] "Generic (PLEG): container finished" podID="55984685-7f0b-473b-a35d-20ad505217a6" containerID="5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d" exitCode=1
Jul 10 19:21:25.926340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:25.926060    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerDied","Data":"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"}
Jul 10 19:21:26.931974 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:26.931948    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/0.log"
Jul 10 19:21:26.932358 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:26.932032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerStarted","Data":"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"}
Jul 10 19:21:30.450021 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:30.449981    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:30.450450 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:30.450219    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer" containerID="cri-o://51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896" gracePeriod=30
Jul 10 19:21:32.290968 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.290943    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/1.log"
Jul 10 19:21:32.291371 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.291356    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/0.log"
Jul 10 19:21:32.293590 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.293575    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:32.305649 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305627    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location\") pod \"55984685-7f0b-473b-a35d-20ad505217a6\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") "
Jul 10 19:21:32.305731 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305680    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config\") pod \"55984685-7f0b-473b-a35d-20ad505217a6\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") "
Jul 10 19:21:32.305731 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305706    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert\") pod \"55984685-7f0b-473b-a35d-20ad505217a6\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") "
Jul 10 19:21:32.305857 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305734    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf\") pod \"55984685-7f0b-473b-a35d-20ad505217a6\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") "
Jul 10 19:21:32.305857 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls\") pod \"55984685-7f0b-473b-a35d-20ad505217a6\" (UID: \"55984685-7f0b-473b-a35d-20ad505217a6\") "
Jul 10 19:21:32.305956 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.305873    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55984685-7f0b-473b-a35d-20ad505217a6" (UID: "55984685-7f0b-473b-a35d-20ad505217a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:32.306022 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.306005    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55984685-7f0b-473b-a35d-20ad505217a6-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.306082 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.306025    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55984685-7f0b-473b-a35d-20ad505217a6" (UID: "55984685-7f0b-473b-a35d-20ad505217a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:32.306082 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.306045    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55984685-7f0b-473b-a35d-20ad505217a6" (UID: "55984685-7f0b-473b-a35d-20ad505217a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:32.307814 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.307792    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf" (OuterVolumeSpecName: "kube-api-access-lhztf") pod "55984685-7f0b-473b-a35d-20ad505217a6" (UID: "55984685-7f0b-473b-a35d-20ad505217a6"). InnerVolumeSpecName "kube-api-access-lhztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:32.307919 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.307831    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55984685-7f0b-473b-a35d-20ad505217a6" (UID: "55984685-7f0b-473b-a35d-20ad505217a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:32.406891 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.406849    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.406891 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.406889    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55984685-7f0b-473b-a35d-20ad505217a6-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.406891 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.406899    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/55984685-7f0b-473b-a35d-20ad505217a6-kube-api-access-lhztf\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.406891 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.406908    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55984685-7f0b-473b-a35d-20ad505217a6-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:21:32.956753 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.956723    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/1.log"
Jul 10 19:21:32.957149 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957132    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv_55984685-7f0b-473b-a35d-20ad505217a6/storage-initializer/0.log"
Jul 10 19:21:32.957221 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957168    2570 generic.go:356] "Generic (PLEG): container finished" podID="55984685-7f0b-473b-a35d-20ad505217a6" containerID="51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896" exitCode=1
Jul 10 19:21:32.957255 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957226    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"
Jul 10 19:21:32.957316 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957224    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerDied","Data":"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"}
Jul 10 19:21:32.957371 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957356    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv" event={"ID":"55984685-7f0b-473b-a35d-20ad505217a6","Type":"ContainerDied","Data":"cde3b83caee1b6ec58b5e3e3c78b1875b032d69ee5ce9a5455d797224f831418"}
Jul 10 19:21:32.957403 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.957397    2570 scope.go:117] "RemoveContainer" containerID="51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"
Jul 10 19:21:32.965359 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.965341    2570 scope.go:117] "RemoveContainer" containerID="5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"
Jul 10 19:21:32.972192 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.972177    2570 scope.go:117] "RemoveContainer" containerID="51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"
Jul 10 19:21:32.972452 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:21:32.972434    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896\": container with ID starting with 51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896 not found: ID does not exist" containerID="51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"
Jul 10 19:21:32.972510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.972460    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896"} err="failed to get container status \"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896\": rpc error: code = NotFound desc = could not find container \"51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896\": container with ID starting with 51d8bd46a3f0f3b1140dea2c53c1db5259ea774ade0dd642e0e7b9ca43f74896 not found: ID does not exist"
Jul 10 19:21:32.972510 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.972476    2570 scope.go:117] "RemoveContainer" containerID="5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"
Jul 10 19:21:32.972715 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:21:32.972698    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d\": container with ID starting with 5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d not found: ID does not exist" containerID="5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"
Jul 10 19:21:32.972757 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.972721    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d"} err="failed to get container status \"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d\": rpc error: code = NotFound desc = could not find container \"5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d\": container with ID starting with 5a2c2c72f9e20235015bbcb09b3118d88c0f5cc39b850cb94b78f63fc4c29e6d not found: ID does not exist"
Jul 10 19:21:32.991215 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.991189    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:32.994584 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:32.994563    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e3c30a-predictor-578df99bf4-drcxv"]
Jul 10 19:21:34.415459 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:34.415419    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55984685-7f0b-473b-a35d-20ad505217a6" path="/var/lib/kubelet/pods/55984685-7f0b-473b-a35d-20ad505217a6/volumes"
Jul 10 19:21:35.668393 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668360    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:21:35.668831 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668735    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.668831 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668750    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.668831 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668760    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.668831 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668782    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.669043 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668844    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.669043 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.668854    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55984685-7f0b-473b-a35d-20ad505217a6" containerName="storage-initializer"
Jul 10 19:21:35.671739 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.671723    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.674516 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.674494    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 10 19:21:35.674630 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.674493    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e7135-kube-rbac-proxy-sar-config\""
Jul 10 19:21:35.674630 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.674493    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 19:21:35.674630 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.674534    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e7135-predictor-serving-cert\""
Jul 10 19:21:35.675429 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.675414    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wct6r\""
Jul 10 19:21:35.675555 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.675535    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:21:35.675751 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.675734    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 19:21:35.681050 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.681032    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:21:35.730957 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.730920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6n5l\" (UniqueName: \"kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.731129 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.730998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.731129 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.731030    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.731129 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.731053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.731129 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.731112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831470 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831415    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831470 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831472    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831671 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6n5l\" (UniqueName: \"kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.831930 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.831907    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.832198 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.832174    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.832317 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.832197    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.833907 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.833884    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.840864 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.840839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6n5l\" (UniqueName: \"kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l\") pod \"raw-sklearn-e7135-predictor-69d867d788-dssrm\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") " pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:35.982891 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:35.982806    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:36.108953 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:36.108842    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:21:36.111070 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:21:36.111048    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b207f5e_df39_431c_9909_47d282555c9b.slice/crio-1294b74b2cfa8b3e79af69c24f8a5404e491e82bf865c0a2dce3f1d701c0d7e7 WatchSource:0}: Error finding container 1294b74b2cfa8b3e79af69c24f8a5404e491e82bf865c0a2dce3f1d701c0d7e7: Status 404 returned error can't find the container with id 1294b74b2cfa8b3e79af69c24f8a5404e491e82bf865c0a2dce3f1d701c0d7e7
Jul 10 19:21:36.976829 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:36.976790    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerStarted","Data":"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"}
Jul 10 19:21:36.977208 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:36.976862    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerStarted","Data":"1294b74b2cfa8b3e79af69c24f8a5404e491e82bf865c0a2dce3f1d701c0d7e7"}
Jul 10 19:21:39.990474 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:39.990392    2570 generic.go:356] "Generic (PLEG): container finished" podID="5b207f5e-df39-431c-9909-47d282555c9b" containerID="ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c" exitCode=0
Jul 10 19:21:39.990837 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:39.990462    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerDied","Data":"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"}
Jul 10 19:21:40.995805 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:40.995752    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerStarted","Data":"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"}
Jul 10 19:21:40.996237 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:40.995828    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerStarted","Data":"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"}
Jul 10 19:21:40.996237 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:40.996012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:41.015625 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:41.015577    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podStartSLOduration=2.491921217 podStartE2EDuration="6.015562741s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.523641524s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:21:41.01519682 +0000 UTC m=+1385.149127488" watchObservedRunningTime="2026-07-10 19:21:41.015562741 +0000 UTC m=+1385.149493408"
Jul 10 19:21:42.000265 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:42.000232    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:42.001629 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:42.001597    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:21:43.004339 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:43.004298    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:21:48.008569 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:48.008535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:21:48.009229 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:48.009198    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:21:58.009661 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:21:58.009610    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:08.009331 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:08.009287    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:18.009790 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:18.009726    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:28.009617 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:28.009529    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:38.009988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:38.009948    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:48.010071 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:48.010028    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:22:58.010478 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:22:58.010448    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:23:05.856697 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:05.856664    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:23:05.857223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:05.856906    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" containerID="cri-o://d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443" gracePeriod=30
Jul 10 19:23:05.857223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:05.856945    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kube-rbac-proxy" containerID="cri-o://7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c" gracePeriod=30
Jul 10 19:23:06.325986 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:06.325952    2570 generic.go:356] "Generic (PLEG): container finished" podID="5b207f5e-df39-431c-9909-47d282555c9b" containerID="7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c" exitCode=2
Jul 10 19:23:06.326149 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:06.326020    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerDied","Data":"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"}
Jul 10 19:23:08.004505 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:08.004465    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 10 19:23:08.009650 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:08.009620    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 10 19:23:10.506729 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.506705    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:23:10.590699 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.590664    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6n5l\" (UniqueName: \"kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l\") pod \"5b207f5e-df39-431c-9909-47d282555c9b\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") "
Jul 10 19:23:10.590918 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.590713    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location\") pod \"5b207f5e-df39-431c-9909-47d282555c9b\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") "
Jul 10 19:23:10.590918 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.590781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls\") pod \"5b207f5e-df39-431c-9909-47d282555c9b\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") "
Jul 10 19:23:10.590918 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.590825    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config\") pod \"5b207f5e-df39-431c-9909-47d282555c9b\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") "
Jul 10 19:23:10.590918 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.590845    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert\") pod \"5b207f5e-df39-431c-9909-47d282555c9b\" (UID: \"5b207f5e-df39-431c-9909-47d282555c9b\") "
Jul 10 19:23:10.591206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.591110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b207f5e-df39-431c-9909-47d282555c9b" (UID: "5b207f5e-df39-431c-9909-47d282555c9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:10.591206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.591199    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b207f5e-df39-431c-9909-47d282555c9b" (UID: "5b207f5e-df39-431c-9909-47d282555c9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:23:10.591312 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.591257    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b207f5e-df39-431c-9909-47d282555c9b" (UID: "5b207f5e-df39-431c-9909-47d282555c9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:23:10.592888 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.592862    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b207f5e-df39-431c-9909-47d282555c9b" (UID: "5b207f5e-df39-431c-9909-47d282555c9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:23:10.592985 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.592926    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l" (OuterVolumeSpecName: "kube-api-access-b6n5l") pod "5b207f5e-df39-431c-9909-47d282555c9b" (UID: "5b207f5e-df39-431c-9909-47d282555c9b"). InnerVolumeSpecName "kube-api-access-b6n5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:23:10.691485 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.691440    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b207f5e-df39-431c-9909-47d282555c9b-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:23:10.691485 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.691478    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:23:10.691485 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.691493    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b207f5e-df39-431c-9909-47d282555c9b-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:23:10.691738 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.691505    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6n5l\" (UniqueName: \"kubernetes.io/projected/5b207f5e-df39-431c-9909-47d282555c9b-kube-api-access-b6n5l\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:23:10.691738 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.691517    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b207f5e-df39-431c-9909-47d282555c9b-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:23:10.986210 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986160    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:23:10.986588 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986572    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="storage-initializer"
Jul 10 19:23:10.986670 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986592    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="storage-initializer"
Jul 10 19:23:10.986670 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986626    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kube-rbac-proxy"
Jul 10 19:23:10.986670 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986637    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kube-rbac-proxy"
Jul 10 19:23:10.986670 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986652    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container"
Jul 10 19:23:10.986670 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986660    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container"
Jul 10 19:23:10.986932 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986737    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kserve-container"
Jul 10 19:23:10.986932 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.986751    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b207f5e-df39-431c-9909-47d282555c9b" containerName="kube-rbac-proxy"
Jul 10 19:23:10.990099 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.990077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:10.992749 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.992722    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-759ec-predictor-serving-cert\""
Jul 10 19:23:10.992870 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:10.992789    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-759ec-kube-rbac-proxy-sar-config\""
Jul 10 19:23:11.001820 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.001795    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:23:11.095102 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.095056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.095289 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.095116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-845vk\" (UniqueName: \"kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.095289 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.095135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.095289 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.095161    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.095289 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.095195    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.195580 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.195536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-845vk\" (UniqueName: \"kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.195580 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.195584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.195855 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.195616    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.195855 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.195658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.195855 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.195724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.196183 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.196160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.196484 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.196463    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.196527 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.196463    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.198084 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.198065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.203862 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.203839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-845vk\" (UniqueName: \"kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk\") pod \"raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.301169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.301059    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:11.349071 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.348985    2570 generic.go:356] "Generic (PLEG): container finished" podID="5b207f5e-df39-431c-9909-47d282555c9b" containerID="d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443" exitCode=0
Jul 10 19:23:11.349071 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.349054    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"
Jul 10 19:23:11.349305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.349114    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerDied","Data":"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"}
Jul 10 19:23:11.349305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.349171    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm" event={"ID":"5b207f5e-df39-431c-9909-47d282555c9b","Type":"ContainerDied","Data":"1294b74b2cfa8b3e79af69c24f8a5404e491e82bf865c0a2dce3f1d701c0d7e7"}
Jul 10 19:23:11.349305 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.349209    2570 scope.go:117] "RemoveContainer" containerID="7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"
Jul 10 19:23:11.366650 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.366622    2570 scope.go:117] "RemoveContainer" containerID="d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"
Jul 10 19:23:11.376472 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.376294    2570 scope.go:117] "RemoveContainer" containerID="ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"
Jul 10 19:23:11.381876 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.380907    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:23:11.388150 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.388122    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e7135-predictor-69d867d788-dssrm"]
Jul 10 19:23:11.393830 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.393553    2570 scope.go:117] "RemoveContainer" containerID="7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"
Jul 10 19:23:11.394036 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:23:11.393990    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c\": container with ID starting with 7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c not found: ID does not exist" containerID="7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"
Jul 10 19:23:11.394152 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.394048    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c"} err="failed to get container status \"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c\": rpc error: code = NotFound desc = could not find container \"7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c\": container with ID starting with 7eadcf20dc1e24f74f6a4e6218ede03da2961e3e30206254e58c35e1a0863f9c not found: ID does not exist"
Jul 10 19:23:11.394152 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.394074    2570 scope.go:117] "RemoveContainer" containerID="d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"
Jul 10 19:23:11.394394 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:23:11.394368    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443\": container with ID starting with d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443 not found: ID does not exist" containerID="d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"
Jul 10 19:23:11.394494 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.394425    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443"} err="failed to get container status \"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443\": rpc error: code = NotFound desc = could not find container \"d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443\": container with ID starting with d4af8adb1553edd0372c0ec1a202d0211aa3776e397ddf65278d9630d5740443 not found: ID does not exist"
Jul 10 19:23:11.394494 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.394447    2570 scope.go:117] "RemoveContainer" containerID="ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"
Jul 10 19:23:11.394810 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:23:11.394757    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c\": container with ID starting with ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c not found: ID does not exist" containerID="ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"
Jul 10 19:23:11.394892 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.394820    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c"} err="failed to get container status \"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c\": rpc error: code = NotFound desc = could not find container \"ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c\": container with ID starting with ed1d3b04eb152d8a6a8a3e3ef5f265deb4b1fada624a73e799eb19ceddc6147c not found: ID does not exist"
Jul 10 19:23:11.457329 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:11.457276    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:23:11.459618 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:23:11.459594    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673e67ee_c25c_47c3_96af_ea55ce890671.slice/crio-16a1cbb647c6acd82bcc1c8fdc482e014b42c19d4d760870fc1487e31c47abf9 WatchSource:0}: Error finding container 16a1cbb647c6acd82bcc1c8fdc482e014b42c19d4d760870fc1487e31c47abf9: Status 404 returned error can't find the container with id 16a1cbb647c6acd82bcc1c8fdc482e014b42c19d4d760870fc1487e31c47abf9
Jul 10 19:23:12.354537 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:12.354505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerStarted","Data":"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"}
Jul 10 19:23:12.354988 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:12.354575    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerStarted","Data":"16a1cbb647c6acd82bcc1c8fdc482e014b42c19d4d760870fc1487e31c47abf9"}
Jul 10 19:23:12.414354 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:12.414317    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b207f5e-df39-431c-9909-47d282555c9b" path="/var/lib/kubelet/pods/5b207f5e-df39-431c-9909-47d282555c9b/volumes"
Jul 10 19:23:15.368375 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:15.368293    2570 generic.go:356] "Generic (PLEG): container finished" podID="673e67ee-c25c-47c3-96af-ea55ce890671" containerID="f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1" exitCode=0
Jul 10 19:23:15.368713 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:15.368366    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerDied","Data":"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"}
Jul 10 19:23:16.373992 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:16.373957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerStarted","Data":"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"}
Jul 10 19:23:16.374468 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:16.374026    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerStarted","Data":"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"}
Jul 10 19:23:16.374468 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:16.374235    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:16.394749 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:16.394681    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podStartSLOduration=2.949131892 podStartE2EDuration="6.394667249s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.445535357s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:23:16.394232686 +0000 UTC m=+1480.528163352" watchObservedRunningTime="2026-07-10 19:23:16.394667249 +0000 UTC m=+1480.528597915"
Jul 10 19:23:17.377787 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:17.377741    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:17.378997 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:17.378968    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:23:18.381365 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:18.381327    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:23:23.386950 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:23.386920    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:23:23.387436 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:23.387409    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:23:33.387836 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:33.387795    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:23:36.385011 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:36.384986    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:23:36.387696 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:36.387673    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:23:43.387544 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:43.387497    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:23:53.387589 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:23:53.387552    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:24:03.387365 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:03.387326    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:24:13.388043 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:13.387998    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:24:23.388293 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:23.388255    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:24:33.387808 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:33.387781    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:24:41.104832 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:41.104799    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:24:41.105247 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:41.105033    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" containerID="cri-o://135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7" gracePeriod=30
Jul 10 19:24:41.105247 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:41.105105    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kube-rbac-proxy" containerID="cri-o://50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47" gracePeriod=30
Jul 10 19:24:41.715784 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:41.715737    2570 generic.go:356] "Generic (PLEG): container finished" podID="673e67ee-c25c-47c3-96af-ea55ce890671" containerID="50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47" exitCode=2
Jul 10 19:24:41.715939 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:41.715806    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerDied","Data":"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"}
Jul 10 19:24:43.382340 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:43.382296    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 10 19:24:43.387675 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:43.387648    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 10 19:24:45.253659 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.253634    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:24:45.323031 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.322974    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-845vk\" (UniqueName: \"kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk\") pod \"673e67ee-c25c-47c3-96af-ea55ce890671\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") "
Jul 10 19:24:45.323031 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323018    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config\") pod \"673e67ee-c25c-47c3-96af-ea55ce890671\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") "
Jul 10 19:24:45.323169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323046    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert\") pod \"673e67ee-c25c-47c3-96af-ea55ce890671\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") "
Jul 10 19:24:45.323169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323080    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location\") pod \"673e67ee-c25c-47c3-96af-ea55ce890671\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") "
Jul 10 19:24:45.323169 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323145    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls\") pod \"673e67ee-c25c-47c3-96af-ea55ce890671\" (UID: \"673e67ee-c25c-47c3-96af-ea55ce890671\") "
Jul 10 19:24:45.323466 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323441    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "673e67ee-c25c-47c3-96af-ea55ce890671" (UID: "673e67ee-c25c-47c3-96af-ea55ce890671"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:24:45.323554 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323438    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "673e67ee-c25c-47c3-96af-ea55ce890671" (UID: "673e67ee-c25c-47c3-96af-ea55ce890671"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:24:45.323554 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.323438    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "673e67ee-c25c-47c3-96af-ea55ce890671" (UID: "673e67ee-c25c-47c3-96af-ea55ce890671"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:24:45.325000 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.324980    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk" (OuterVolumeSpecName: "kube-api-access-845vk") pod "673e67ee-c25c-47c3-96af-ea55ce890671" (UID: "673e67ee-c25c-47c3-96af-ea55ce890671"). InnerVolumeSpecName "kube-api-access-845vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:24:45.325119 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.325097    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "673e67ee-c25c-47c3-96af-ea55ce890671" (UID: "673e67ee-c25c-47c3-96af-ea55ce890671"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:24:45.423963 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.423934    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/673e67ee-c25c-47c3-96af-ea55ce890671-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:24:45.423963 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.423962    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-845vk\" (UniqueName: \"kubernetes.io/projected/673e67ee-c25c-47c3-96af-ea55ce890671-kube-api-access-845vk\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:24:45.423963 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.423971    2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:24:45.424120 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.423980    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/673e67ee-c25c-47c3-96af-ea55ce890671-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:24:45.424120 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.423988    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/673e67ee-c25c-47c3-96af-ea55ce890671-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:24:45.731303 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.731272    2570 generic.go:356] "Generic (PLEG): container finished" podID="673e67ee-c25c-47c3-96af-ea55ce890671" containerID="135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7" exitCode=0
Jul 10 19:24:45.731471 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.731314    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerDied","Data":"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"}
Jul 10 19:24:45.731471 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.731334    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"
Jul 10 19:24:45.731471 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.731362    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws" event={"ID":"673e67ee-c25c-47c3-96af-ea55ce890671","Type":"ContainerDied","Data":"16a1cbb647c6acd82bcc1c8fdc482e014b42c19d4d760870fc1487e31c47abf9"}
Jul 10 19:24:45.731471 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.731411    2570 scope.go:117] "RemoveContainer" containerID="50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"
Jul 10 19:24:45.739491 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.739471    2570 scope.go:117] "RemoveContainer" containerID="135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"
Jul 10 19:24:45.746565 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.746545    2570 scope.go:117] "RemoveContainer" containerID="f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"
Jul 10 19:24:45.753926 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.753906    2570 scope.go:117] "RemoveContainer" containerID="50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"
Jul 10 19:24:45.754025 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754005    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:24:45.754168 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:24:45.754150    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47\": container with ID starting with 50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47 not found: ID does not exist" containerID="50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"
Jul 10 19:24:45.754236 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754181    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47"} err="failed to get container status \"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47\": rpc error: code = NotFound desc = could not find container \"50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47\": container with ID starting with 50793bbf1a18eca0675beb38d0643ab11e9e8a82922b973fd3243a5cd113af47 not found: ID does not exist"
Jul 10 19:24:45.754236 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754205    2570 scope.go:117] "RemoveContainer" containerID="135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"
Jul 10 19:24:45.754438 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:24:45.754421    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7\": container with ID starting with 135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7 not found: ID does not exist" containerID="135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"
Jul 10 19:24:45.754486 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754445    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7"} err="failed to get container status \"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7\": rpc error: code = NotFound desc = could not find container \"135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7\": container with ID starting with 135692adfe44dc4f69fd77a8e9aefe871e0ac4e6767e2545f64ea11cae7632d7 not found: ID does not exist"
Jul 10 19:24:45.754486 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754459    2570 scope.go:117] "RemoveContainer" containerID="f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"
Jul 10 19:24:45.754678 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:24:45.754662    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1\": container with ID starting with f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1 not found: ID does not exist" containerID="f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"
Jul 10 19:24:45.754723 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.754684    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1"} err="failed to get container status \"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1\": rpc error: code = NotFound desc = could not find container \"f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1\": container with ID starting with f21e547e258d551110846a2028f2345cbaa7b6869481ef870360948aebefbaa1 not found: ID does not exist"
Jul 10 19:24:45.756934 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:45.756915    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-759ec-predictor-547bc74f8d-lsdws"]
Jul 10 19:24:46.413937 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:46.413906    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" path="/var/lib/kubelet/pods/673e67ee-c25c-47c3-96af-ea55ce890671/volumes"
Jul 10 19:24:47.646534 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646506    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgbdx/must-gather-t9d5d"]
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646794    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="storage-initializer"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646805    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="storage-initializer"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646823    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646828    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646837    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kube-rbac-proxy"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646844    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kube-rbac-proxy"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646889    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kserve-container"
Jul 10 19:24:47.646906 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.646898    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="673e67ee-c25c-47c3-96af-ea55ce890671" containerName="kube-rbac-proxy"
Jul 10 19:24:47.651243 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.651217    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.654056 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.654035    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgbdx\"/\"openshift-service-ca.crt\""
Jul 10 19:24:47.655132 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.655117    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rgbdx\"/\"default-dockercfg-75x9b\""
Jul 10 19:24:47.655202 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.655121    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgbdx\"/\"kube-root-ca.crt\""
Jul 10 19:24:47.658163 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.657997    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbdx/must-gather-t9d5d"]
Jul 10 19:24:47.737940 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.737915    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvz5\" (UniqueName: \"kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.738056 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.737956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.839251 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.839227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvz5\" (UniqueName: \"kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.839364 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.839270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.839533 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.839518    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.847870 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.847843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvz5\" (UniqueName: \"kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5\") pod \"must-gather-t9d5d\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") " pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:47.973376 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:47.973315    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:24:48.092487 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:48.092463    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgbdx/must-gather-t9d5d"]
Jul 10 19:24:48.094846 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:24:48.094819    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea60d7e5_d4cd_4da4_91a2_f201de5f0642.slice/crio-f0a281ceb15b245dc481107ba9d8472031b670ae1a4f411cf2b22dd3c3a7199e WatchSource:0}: Error finding container f0a281ceb15b245dc481107ba9d8472031b670ae1a4f411cf2b22dd3c3a7199e: Status 404 returned error can't find the container with id f0a281ceb15b245dc481107ba9d8472031b670ae1a4f411cf2b22dd3c3a7199e
Jul 10 19:24:48.096633 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:48.096618    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:24:48.744197 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:48.744167    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" event={"ID":"ea60d7e5-d4cd-4da4-91a2-f201de5f0642","Type":"ContainerStarted","Data":"f0a281ceb15b245dc481107ba9d8472031b670ae1a4f411cf2b22dd3c3a7199e"}
Jul 10 19:24:54.773457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:54.773411    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" event={"ID":"ea60d7e5-d4cd-4da4-91a2-f201de5f0642","Type":"ContainerStarted","Data":"cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09"}
Jul 10 19:24:54.773457 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:54.773457    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" event={"ID":"ea60d7e5-d4cd-4da4-91a2-f201de5f0642","Type":"ContainerStarted","Data":"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"}
Jul 10 19:24:54.789317 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:24:54.789271    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" podStartSLOduration=1.69760298 podStartE2EDuration="7.789257023s" totalImagesPullingTime="6.091654043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:54.788254561 +0000 UTC m=+1578.922185210" watchObservedRunningTime="2026-07-10 19:24:54.789257023 +0000 UTC m=+1578.923187690"
Jul 10 19:25:14.867862 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:14.867757    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerID="2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690" exitCode=0
Jul 10 19:25:14.867862 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:14.867831    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" event={"ID":"ea60d7e5-d4cd-4da4-91a2-f201de5f0642","Type":"ContainerDied","Data":"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"}
Jul 10 19:25:14.868300 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:14.868139    2570 scope.go:117] "RemoveContainer" containerID="2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"
Jul 10 19:25:15.588949 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:15.588928    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rgbdx_must-gather-t9d5d_ea60d7e5-d4cd-4da4-91a2-f201de5f0642/gather/0.log"
Jul 10 19:25:18.909887 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:18.909853    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6d6ql_17b396ae-7bc9-41ef-8a1c-5a2e10de647a/global-pull-secret-syncer/0.log"
Jul 10 19:25:19.090645 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:19.090614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g5rzb_d6be89d1-7d75-49fd-b6fb-23a749d08caf/konnectivity-agent/0.log"
Jul 10 19:25:19.191218 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:19.191146    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-26.ec2.internal_a274b38950da356e83a1d1b30ba640f1/haproxy/0.log"
Jul 10 19:25:21.023619 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.023586    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rgbdx/must-gather-t9d5d"]
Jul 10 19:25:21.024131 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.023796    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rgbdx/must-gather-t9d5d" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="copy" containerID="cri-o://cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09" gracePeriod=2
Jul 10 19:25:21.029964 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.029939    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rgbdx/must-gather-t9d5d"]
Jul 10 19:25:21.030372 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.030339    2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-rgbdx\"/\"must-gather-t9d5d\": pods \"must-gather-t9d5d\" not found" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.244895 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.244871    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rgbdx_must-gather-t9d5d_ea60d7e5-d4cd-4da4-91a2-f201de5f0642/copy/0.log"
Jul 10 19:25:21.246512 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.246492    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.248694 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.248671    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9d5d\" is forbidden: User \"system:node:ip-10-0-134-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rgbdx\": no relationship found between node 'ip-10-0-134-26.ec2.internal' and this object" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.406711 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.406677    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output\") pod \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") "
Jul 10 19:25:21.406870 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.406722    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjvz5\" (UniqueName: \"kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5\") pod \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\" (UID: \"ea60d7e5-d4cd-4da4-91a2-f201de5f0642\") "
Jul 10 19:25:21.408081 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.408052    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ea60d7e5-d4cd-4da4-91a2-f201de5f0642" (UID: "ea60d7e5-d4cd-4da4-91a2-f201de5f0642"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:25:21.408741 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.408721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5" (OuterVolumeSpecName: "kube-api-access-pjvz5") pod "ea60d7e5-d4cd-4da4-91a2-f201de5f0642" (UID: "ea60d7e5-d4cd-4da4-91a2-f201de5f0642"). InnerVolumeSpecName "kube-api-access-pjvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:25:21.508206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.508177    2570 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-must-gather-output\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:25:21.508206 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.508205    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvz5\" (UniqueName: \"kubernetes.io/projected/ea60d7e5-d4cd-4da4-91a2-f201de5f0642-kube-api-access-pjvz5\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Jul 10 19:25:21.895777 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.895744    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rgbdx_must-gather-t9d5d_ea60d7e5-d4cd-4da4-91a2-f201de5f0642/copy/0.log"
Jul 10 19:25:21.896123 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.896102    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerID="cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09" exitCode=143
Jul 10 19:25:21.896222 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.896157    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.896222 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.896208    2570 scope.go:117] "RemoveContainer" containerID="cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09"
Jul 10 19:25:21.898741 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.898710    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9d5d\" is forbidden: User \"system:node:ip-10-0-134-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rgbdx\": no relationship found between node 'ip-10-0-134-26.ec2.internal' and this object" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.904738 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.904720    2570 scope.go:117] "RemoveContainer" containerID="2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"
Jul 10 19:25:21.906984 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.906958    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9d5d\" is forbidden: User \"system:node:ip-10-0-134-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rgbdx\": no relationship found between node 'ip-10-0-134-26.ec2.internal' and this object" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" pod="openshift-must-gather-rgbdx/must-gather-t9d5d"
Jul 10 19:25:21.915522 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.915507    2570 scope.go:117] "RemoveContainer" containerID="cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09"
Jul 10 19:25:21.915757 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.915735    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09\": container with ID starting with cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09 not found: ID does not exist" containerID="cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09"
Jul 10 19:25:21.915878 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.915785    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09"} err="failed to get container status \"cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09\": rpc error: code = NotFound desc = could not find container \"cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09\": container with ID starting with cea28b8d7475c56eb26f279bf5c19818436854ce28d14166623396e99d725a09 not found: ID does not exist"
Jul 10 19:25:21.915878 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.915808    2570 scope.go:117] "RemoveContainer" containerID="2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"
Jul 10 19:25:21.916073 ip-10-0-134-26 kubenswrapper[2570]: E0710 19:25:21.916057    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690\": container with ID starting with 2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690 not found: ID does not exist" containerID="2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"
Jul 10 19:25:21.916110 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:21.916080    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690"} err="failed to get container status \"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690\": rpc error: code = NotFound desc = could not find container \"2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690\": container with ID starting with 2058fcc2ad9ff08eb5bfc3f9092cb575c61462cadcdf5518a1783c5e0eea5690 not found: ID does not exist"
Jul 10 19:25:22.414589 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:22.414557    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" path="/var/lib/kubelet/pods/ea60d7e5-d4cd-4da4-91a2-f201de5f0642/volumes"
Jul 10 19:25:24.194027 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.194004    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-98f2r_2328632a-950d-48a8-a941-5bc1e01090b8/node-exporter/0.log"
Jul 10 19:25:24.222897 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.222870    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-98f2r_2328632a-950d-48a8-a941-5bc1e01090b8/kube-rbac-proxy/0.log"
Jul 10 19:25:24.246492 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.246472    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-98f2r_2328632a-950d-48a8-a941-5bc1e01090b8/init-textfile/0.log"
Jul 10 19:25:24.730225 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.730199    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hnhtd_e6400aba-269b-456a-a1a5-13b37504ecbc/prometheus-operator/0.log"
Jul 10 19:25:24.750749 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.750722    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hnhtd_e6400aba-269b-456a-a1a5-13b37504ecbc/kube-rbac-proxy/0.log"
Jul 10 19:25:24.778038 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.778017    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-xmgwj_bf7a3746-41b3-4385-a1e3-3d38ae093ad1/prometheus-operator-admission-webhook/0.log"
Jul 10 19:25:24.904266 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.904235    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/thanos-query/0.log"
Jul 10 19:25:24.929692 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.929665    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/kube-rbac-proxy-web/0.log"
Jul 10 19:25:24.959317 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.959297    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/kube-rbac-proxy/0.log"
Jul 10 19:25:24.988383 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:24.988328    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/prom-label-proxy/0.log"
Jul 10 19:25:25.014423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:25.014404    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/kube-rbac-proxy-rules/0.log"
Jul 10 19:25:25.042075 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:25.042053    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-674d8c9cbb-m9xgd_4605d126-eb55-4e5c-83e7-1f94eaf58f04/kube-rbac-proxy-metrics/0.log"
Jul 10 19:25:27.278826 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.278794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-v965r_62f063d8-32d1-4393-9c83-06f9a37f23af/download-server/0.log"
Jul 10 19:25:27.514541 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514514    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"]
Jul 10 19:25:27.514818 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514803    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="copy"
Jul 10 19:25:27.514883 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514820    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="copy"
Jul 10 19:25:27.514883 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514843    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="gather"
Jul 10 19:25:27.514883 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514850    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="gather"
Jul 10 19:25:27.514977 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514909    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="copy"
Jul 10 19:25:27.514977 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.514918    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea60d7e5-d4cd-4da4-91a2-f201de5f0642" containerName="gather"
Jul 10 19:25:27.518458 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.518439    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.520960 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.520942    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r2nq6\"/\"default-dockercfg-vzxsr\""
Jul 10 19:25:27.522100 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.522079    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2nq6\"/\"openshift-service-ca.crt\""
Jul 10 19:25:27.522188 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.522082    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2nq6\"/\"kube-root-ca.crt\""
Jul 10 19:25:27.528228 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.528207    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"]
Jul 10 19:25:27.551837 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.551778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-sys\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.551936 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.551864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-proc\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.551936 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.551909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl69d\" (UniqueName: \"kubernetes.io/projected/50314025-0a71-4179-88dd-65c1787d51f0-kube-api-access-bl69d\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.552042 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.551937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-podres\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.552042 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.551964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-lib-modules\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-podres\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652326 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-lib-modules\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652326 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-sys\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652326 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-proc\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl69d\" (UniqueName: \"kubernetes.io/projected/50314025-0a71-4179-88dd-65c1787d51f0-kube-api-access-bl69d\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652370    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-podres\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652380    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-lib-modules\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652388    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-sys\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.652455 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.652405    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50314025-0a71-4179-88dd-65c1787d51f0-proc\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.660834 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.660810    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl69d\" (UniqueName: \"kubernetes.io/projected/50314025-0a71-4179-88dd-65c1787d51f0-kube-api-access-bl69d\") pod \"perf-node-gather-daemonset-9dtl9\" (UID: \"50314025-0a71-4179-88dd-65c1787d51f0\") " pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.828382 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.828328    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:27.955060 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:27.955036    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"]
Jul 10 19:25:27.957648 ip-10-0-134-26 kubenswrapper[2570]: W0710 19:25:27.957624    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50314025_0a71_4179_88dd_65c1787d51f0.slice/crio-e09b750c59b07a8cbc437ea557eac6247a68cdcc8ffbbf55a70070f52e2244b4 WatchSource:0}: Error finding container e09b750c59b07a8cbc437ea557eac6247a68cdcc8ffbbf55a70070f52e2244b4: Status 404 returned error can't find the container with id e09b750c59b07a8cbc437ea557eac6247a68cdcc8ffbbf55a70070f52e2244b4
Jul 10 19:25:28.782119 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.782093    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mllg9_4d9bdc1d-3a17-4970-992f-fc348fd4a8e1/dns/0.log"
Jul 10 19:25:28.804939 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.804902    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mllg9_4d9bdc1d-3a17-4970-992f-fc348fd4a8e1/kube-rbac-proxy/0.log"
Jul 10 19:25:28.875968 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.875949    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsggr_3179a044-43c4-48c9-9175-810563d50e49/dns-node-resolver/0.log"
Jul 10 19:25:28.931154 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.931124    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9" event={"ID":"50314025-0a71-4179-88dd-65c1787d51f0","Type":"ContainerStarted","Data":"9af3d0e33ec30149f2e02c6c275ac09e8f7144f77241a3ebcc6aa3bbaed934bc"}
Jul 10 19:25:28.931310 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.931194    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9" event={"ID":"50314025-0a71-4179-88dd-65c1787d51f0","Type":"ContainerStarted","Data":"e09b750c59b07a8cbc437ea557eac6247a68cdcc8ffbbf55a70070f52e2244b4"}
Jul 10 19:25:28.931310 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.931270    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:28.948572 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:28.948533    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9" podStartSLOduration=1.948521558 podStartE2EDuration="1.948521558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:28.94748017 +0000 UTC m=+1613.081410831" watchObservedRunningTime="2026-07-10 19:25:28.948521558 +0000 UTC m=+1613.082452224"
Jul 10 19:25:29.318996 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:29.318966    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d7fc75cb8-46vdg_15bd5471-1d37-403d-bea8-db0218335df7/registry/0.log"
Jul 10 19:25:29.364145 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:29.364121    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcb4c_3783f4ff-7409-46f9-9c40-7beea2ada6c0/node-ca/0.log"
Jul 10 19:25:30.196806 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:30.196776    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-648566b76d-5cwnc_846ea7b0-fbdc-4a3c-93d7-be041d80210a/router/0.log"
Jul 10 19:25:30.611205 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:30.611125    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-km6xx_fdca6c0c-efeb-4cae-876c-e9e55d0c24a3/serve-healthcheck-canary/0.log"
Jul 10 19:25:31.000018 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:30.999990    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-jhgc6_256d0b26-9b34-46d7-9b6e-e05911b13b66/insights-operator/1.log"
Jul 10 19:25:31.000279 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:31.000259    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-jhgc6_256d0b26-9b34-46d7-9b6e-e05911b13b66/insights-operator/0.log"
Jul 10 19:25:31.171855 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:31.171832    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-blkm7_bc1193ca-6386-46ad-9f2f-0acc30d02996/kube-rbac-proxy/0.log"
Jul 10 19:25:31.191602 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:31.191581    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-blkm7_bc1193ca-6386-46ad-9f2f-0acc30d02996/exporter/0.log"
Jul 10 19:25:31.212485 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:31.212466    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-blkm7_bc1193ca-6386-46ad-9f2f-0acc30d02996/extractor/0.log"
Jul 10 19:25:33.307402 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:33.307374    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-bl9td_0564b586-91e6-46b8-8644-6e3fc61ce939/server/0.log"
Jul 10 19:25:33.430488 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:33.430451    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-g9z8q_0067a7a1-b862-4014-9265-747567d95aef/manager/0.log"
Jul 10 19:25:33.472434 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:33.472411    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mvjtr_9ca0a5de-8047-41f9-b6f6-7e84f9e244f0/s3-init/0.log"
Jul 10 19:25:33.489051 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:33.489032    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mvjtr_9ca0a5de-8047-41f9-b6f6-7e84f9e244f0/download-hf-model/0.log"
Jul 10 19:25:33.518920 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:33.518899    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nqljj_817feb75-ad27-4329-b412-cf087beb1c99/seaweedfs/0.log"
Jul 10 19:25:34.943937 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:34.943911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r2nq6/perf-node-gather-daemonset-9dtl9"
Jul 10 19:25:37.581255 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:37.581221    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n5rrx_ce49190c-cda3-4a90-a536-ec1a35e98730/migrator/0.log"
Jul 10 19:25:37.604217 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:37.604198    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n5rrx_ce49190c-cda3-4a90-a536-ec1a35e98730/graceful-termination/0.log"
Jul 10 19:25:39.229174 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.229146    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:25:39.251536 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.251514    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/egress-router-binary-copy/0.log"
Jul 10 19:25:39.274486 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.274465    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/cni-plugins/0.log"
Jul 10 19:25:39.301186 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.301162    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/bond-cni-plugin/0.log"
Jul 10 19:25:39.322733 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.322710    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/routeoverride-cni/0.log"
Jul 10 19:25:39.344605 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.344582    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/whereabouts-cni-bincopy/0.log"
Jul 10 19:25:39.365456 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.365441    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m2xd5_b226814b-8a33-46dc-af68-bafeb83fa909/whereabouts-cni/0.log"
Jul 10 19:25:39.404714 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.404694    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hcf89_2a57bc5d-8e88-42b6-8fdf-2f262c61125a/kube-multus/0.log"
Jul 10 19:25:39.508407 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.508336    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8qdk8_a0f255b1-7837-4251-9bca-3aa6563d4333/network-metrics-daemon/0.log"
Jul 10 19:25:39.527223 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:39.527204    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8qdk8_a0f255b1-7837-4251-9bca-3aa6563d4333/kube-rbac-proxy/0.log"
Jul 10 19:25:40.526715 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.526663    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-controller/0.log"
Jul 10 19:25:40.545273 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.545247    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/1.log"
Jul 10 19:25:40.545423 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.545342    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovn-acl-logging/0.log"
Jul 10 19:25:40.565064 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.565043    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/kube-rbac-proxy-node/0.log"
Jul 10 19:25:40.585161 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.585128    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:25:40.601917 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.601894    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/northd/0.log"
Jul 10 19:25:40.624203 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.624180    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/nbdb/0.log"
Jul 10 19:25:40.649732 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.649714    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/sbdb/0.log"
Jul 10 19:25:40.704965 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:40.704945    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5mrx_9a1b3146-0f9e-49f0-977d-8887ff75c7a4/ovnkube-controller/0.log"
Jul 10 19:25:41.937439 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:41.937413    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jg5tw_07e43abd-974e-4a92-9898-e886edee0be1/network-check-target-container/0.log"
Jul 10 19:25:42.926699 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:42.926674    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9496x_c6d1484d-d443-4c20-ae86-5eeb82fc60b4/iptables-alerter/0.log"
Jul 10 19:25:43.638724 ip-10-0-134-26 kubenswrapper[2570]: I0710 19:25:43.638688    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nfh97_8ab21d36-15af-4811-b62b-0acee70cee18/tuned/0.log"