Jul 02 13:13:06.174406 ip-10-0-134-102 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 13:13:06.174418 ip-10-0-134-102 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 13:13:06.174424 ip-10-0-134-102 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 13:13:06.174663 ip-10-0-134-102 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 13:13:16.424409 ip-10-0-134-102 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 13:13:16.424433 ip-10-0-134-102 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 40a87485fbd74873b51bee5bbf272357 --
Jul 02 13:14:52.814311 ip-10-0-134-102 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.126463 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.124724    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129692    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129705    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129708    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129711    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129716    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129719    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129721    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129724    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129727    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129729    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129732    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129735    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129738    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129740    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129743    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129746    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129749    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129751    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129754    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.305535 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129762    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.197869 ip-10-0-134-102 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129765    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129768    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129770    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129773    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129776    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129780    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129784    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129787    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129789    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129792    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129795    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129798    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129801    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129803    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129806    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129808    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129811    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129814    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.319360 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129817    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129819    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129822    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129824    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129827    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129829    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129832    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129834    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129837    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129839    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129842    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129847    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129850    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129853    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129856    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129858    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129861    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129863    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129866    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129869    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.320474 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129871    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129874    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129876    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129879    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129882    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129885    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129887    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129890    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129892    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129895    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129897    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129900    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129902    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129905    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129908    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129910    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129913    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129915    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129918    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129920    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.321580 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129922    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129926    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129929    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129934    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129937    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129940    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129943    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129946    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129949    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129952    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129954    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129957    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129960    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129962    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129965    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129969    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129972    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129975    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129978    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129980    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.322219 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.129983    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130681    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130688    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130691    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130694    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130696    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130700    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130703    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130706    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130708    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130711    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130714    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130718    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130722    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130725    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130728    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130732    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130735    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130738    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.322821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130741    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130743    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130746    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130749    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130751    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130754    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130757    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130760    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130762    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130765    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130768    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130771    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130775    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130777    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130780    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130782    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130785    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130788    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130791    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130794    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.323593 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130797    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130801    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130804    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130807    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130810    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130812    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130815    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130818    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130820    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130823    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130827    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130830    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130834    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130836    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130839    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130842    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130844    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130848    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130850    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.324230 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130853    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130855    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130858    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130861    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130864    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130866    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130869    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130872    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130874    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130876    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130880    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130883    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130886    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130888    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130891    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130894    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130897    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130902    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130904    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.324869 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130906    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130909    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130911    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130914    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130916    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130919    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130921    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130924    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130926    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130929    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130932    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130935    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130937    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130940    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130942    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130945    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130947    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130950    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130952    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130955    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.325558 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130958    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.130960    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131326    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131331    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131334    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131338    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131343    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131346    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131349    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131351    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131354    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131357    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131360    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131363    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131365    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131368    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131371    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131373    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131376    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131379    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.326245 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131382    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131384    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131387    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131390    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131393    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131395    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131398    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131401    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131405    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131408    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131410    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131413    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131415    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131418    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131420    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131425    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131428    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131430    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131433    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.326867 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131436    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131438    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131441    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131443    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131446    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131448    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131451    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131453    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131456    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131459    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131461    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131464    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131466    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131469    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131471    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131474    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131476    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131479    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131482    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131484    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.327446 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131487    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131489    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131492    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131494    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131497    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131499    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131502    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131505    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131508    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131511    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131513    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131516    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131519    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131523    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131526    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131528    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131531    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131535    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131539    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.328001 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131542    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131545    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131548    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131550    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131553    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131555    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131557    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131560    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131563    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131565    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131568    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131570    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131573    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131575    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131577    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131580    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131582    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131585    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131587    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131590    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.328602 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131593    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131596    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.131599    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131666    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131673    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131678    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131684    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131688    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131692    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131696    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131700    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131703    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131706    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131709    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131712    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131715    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131718    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131721    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131724    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131727    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131730    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131734    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131737    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131740    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:14:53.329234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131743    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131747    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131751    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131754    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131757    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131760    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131763    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131765    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131768    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131772    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131775    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131779    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131782    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131785    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131789    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131793    2565 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131796    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131800    2565 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131803    2565 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131806    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131809    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131812    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131815    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131818    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131821    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:14:53.329940 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131824    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131827    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131830    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131832    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131835    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131838    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131841    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131844    2565 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131847    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131850    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131853    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131856    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131859    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131862    2565 flags.go:64] FLAG: --help="false"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131865    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-134-102.ec2.internal"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131868    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131872    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131875    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131878    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131881    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131884    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131889    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131892    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:14:53.330753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131895    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131898    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131901    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131904    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131907    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131910    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131913    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131915    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131918    2565 flags.go:64] FLAG: --lock-file=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131921    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131923    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131926    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131931    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131934    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131937    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131940    2565 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131942    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131946    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131948    2565 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131951    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131955    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131958    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131962    2565 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131964    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131967    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:14:53.331580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131971    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131974    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131977    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131980    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131983    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131991    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131994    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.131997    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132000    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132010    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132015    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132018    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132021    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132024    2565 flags.go:64] FLAG: --port="10250"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132027    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132043    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e2fee35d6ad719a9"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132046    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132049    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132052    2565 flags.go:64] FLAG: --register-node="true"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132055    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132063    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132066    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132069    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132072    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132075    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:14:53.332373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132078    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132081    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132084    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132087    2565 flags.go:64] FLAG: --runonce="false"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132090    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132093    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132096    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132099    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132103    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132106    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132109    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132112    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132116    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132119    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132122    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132124    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132127    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132131    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132134    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132139    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132142    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132145    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132149    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132152    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132155    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:14:53.333013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132157    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132160    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132163    2565 flags.go:64] FLAG: --v="2"
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132167    2565 flags.go:64] FLAG: --version="false"
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132171    2565 flags.go:64] FLAG: --vmodule=""
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132174    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.132177    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132283    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132287    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132290    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132293    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132296    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132299    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132301    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132304    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132308    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132311    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132313    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132316    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132320    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132322    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.333733 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132325    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132329    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132333    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132335    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132339    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132341    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132344    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132346    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132349    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132352    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132354    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132357    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132360    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132362    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132365    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132367    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132370    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132373    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132375    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.334538 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132378    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132380    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132383    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132385    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132388    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132390    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132393    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132396    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132399    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132401    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132404    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132407    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132410    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132413    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132415    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132418    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132420    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132423    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132427    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132430    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.335097 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132434    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132437    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132440    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132443    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132445    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132448    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132451    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132453    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132456    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132458    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132461    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132463    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132467    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132469    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132471    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132474    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132477    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132479    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132482    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132486    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:54.950895 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132488    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132490    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132493    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132497    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132517    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132522    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132525    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132528    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132531    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132534    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132536    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132539    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132542    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132544    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132547    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132549    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132552    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132555    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132557    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:54.951879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132560    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132562    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132565    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132567    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132570    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132572    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.132575    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.133148    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.140383    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.140398    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.141071    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.141079    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141135    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141139    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141142    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:54.952610 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141146    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141149    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141152    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141156    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141160    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141164    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141167    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141170    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141172    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141175    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141178    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141181    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141183    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141186    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141189    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141192    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141195    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141198    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141201    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:54.953136 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141203    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141206    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141209    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141211    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141214    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141218    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141220    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141223    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141226    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141228    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141231    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141234    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141236    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141239    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141242    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141244    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141247    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141249    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141252    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141254    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:54.953662 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141257    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141259    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141262    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141264    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141267    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141270    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141274    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141276    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141279    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141282    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141284    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141287    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141289    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141291    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141294    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141297    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141299    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141302    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141305    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141308    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:54.954266 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141310    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141313    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141316    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141320    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141323    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141325    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141328    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141330    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141333    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141336    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141338    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141341    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141344    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141347    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141349    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141351    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141354    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141357    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141360    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141363    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:54.954817 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141365    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141370    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141373    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141376    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141379    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141381    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141384    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141386    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141389    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141392    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141395    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141398    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141400    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141403    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141405    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141408    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:54.955498 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.141410    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.141416    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145060    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145074    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145077    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145080    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145083    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145086    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145089    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145092    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145095    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145097    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145100    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145103    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145106    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:54.955973 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145109    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145114    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145118    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145121    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145124    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145127    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145130    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145133    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145136    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145138    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145141    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145143    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145146    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145148    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145151    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145154    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145156    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145159    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145162    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:54.956380 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145166    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145169    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145171    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145174    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145177    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145179    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145182    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145185    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145187    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145190    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145192    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145195    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145197    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145201    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145203    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145206    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145208    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145211    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145213    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145216    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:54.956898 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145219    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145221    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145224    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145226    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145229    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145231    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145234    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145236    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145238    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145241    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145244    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145246    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145249    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145253    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145255    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145258    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145260    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145263    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145266    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:54.957490 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145269    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145272    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145275    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145277    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145280    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145283    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145285    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145289    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145291    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145294    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145296    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145299    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145302    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145305    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145307    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145309    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145312    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145314    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145317    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145320    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:54.958028 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145322    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145325    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145328    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145330    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145333    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145341    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145344    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:53.145348    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.145354    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.145457    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.149768    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.150974    2565 server.go:1019] "Starting client certificate rotation"
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.151058    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:54.958600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.151092    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.153100    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.156449    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.167829    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.169878    2565 log.go:25] "Validated CRI v1 image API"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.170720    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.172730    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac5d98c7-f9c3-4a1e-8138-d6d95493fe6e:/dev/nvme0n1p4 cabc5bc8-eb96-4c67-9047-7049c634b75c:/dev/nvme0n1p3]
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.172743    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:14:54.959013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.180415    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.181134    2565 manager.go:216] Machine: {Timestamp:2026-07-02 13:14:53.18065881 +0000 UTC m=+0.272368200 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088919 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c4998cc11f9b65841aabe694edf6 SystemUUID:ec26c499-8cc1-1f9b-6584-1aabe694edf6 BootID:40a87485-fbd7-4873-b51b-ee5bbf272357 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:9f:ae:71:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:9f:ae:71:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:7d:e4:eb:9e:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.181222    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.181290    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.183731    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.183755    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.184398    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.184407    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.184415    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.184428    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.185253    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.188936    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189069    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189096    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189105    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189118    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189126    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189135    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:14:54.959402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189425    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.189439    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.191205    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.192479    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.192505    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193101    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193113    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193119    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193124    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193133    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193139    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193147    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193153    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193159    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193164    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193173    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193179    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193780    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.193787    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.197023    2565 server.go:1268] "Started kubelet"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.197115    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.197140    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.197211    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.198417    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.198902    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.200528    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.200629    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.200641    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.200734    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-102.ec2.internal.18be7b51ac34b51b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-102.ec2.internal,UID:ip-10-0-134-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-102.ec2.internal,},FirstTimestamp:2026-07-02 13:14:53.196997915 +0000 UTC m=+0.288707305,LastTimestamp:2026-07-02 13:14:53.196997915 +0000 UTC m=+0.288707305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-102.ec2.internal,}"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.203446    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9pkfp"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.203601    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.204971    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.204988    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.205054    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.205515    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.205540    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.205555    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.205636    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:14:54.960047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.205644    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.206986    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.209900    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210052    2565 factory.go:55] Registering systemd factory
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210064    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210096    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9pkfp"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210274    2565 factory.go:153] Registering CRI-O factory
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210287    2565 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210331    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210349    2565 factory.go:103] Registering Raw factory
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210360    2565 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.210785    2565 manager.go:349] Starting recovery of all containers
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.223168    2565 manager.go:354] Recovery completed
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.229738    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.231410    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.231439    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.231451    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.231951    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.231952    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.232003    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.232019    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.232562    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.232585    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.232594    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.234101    2565 policy_none.go:49] "None policy: Start"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.234113    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.234123    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.236098    2565 policy_none.go:47] "Start"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.278166    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.279337    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.279384    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.279411    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.279451    2565 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.280825    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.281589    2565 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.281614    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.281625    2565 server.go:87] "Starting device plugin registration server"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.281899    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.281912    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.281979    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.282085    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.282093    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.282931    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.282974    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.380102    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"]
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.380180    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.142414 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.381291    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.381321    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.381334    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.382436    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383246    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383274    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383286    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383309    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383652    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.383779    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384267    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384285    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384313    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384322    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384292    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.384375    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.386517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.386536    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.387160    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.387183    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.387194    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:53.391968    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.391984    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-102.ec2.internal\": node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.413204    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:53.415835    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-102.ec2.internal\" not found" node="ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:54.842431    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-102.ec2.internal\" not found" node="ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:54.929206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:54.929233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:54.929249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67cd314639f46c104191661c02cd9b02-config\") pod \"kube-apiserver-proxy-ip-10-0-134-102.ec2.internal\" (UID: \"67cd314639f46c104191661c02cd9b02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:54.935205    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.144107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030156    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67cd314639f46c104191661c02cd9b02-config\") pod \"kube-apiserver-proxy-ip-10-0-134-102.ec2.internal\" (UID: \"67cd314639f46c104191661c02cd9b02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030184    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67cd314639f46c104191661c02cd9b02-config\") pod \"kube-apiserver-proxy-ip-10-0-134-102.ec2.internal\" (UID: \"67cd314639f46c104191661c02cd9b02\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.030221    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/543120c32551afd3c62e5d553d438de2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal\" (UID: \"543120c32551afd3c62e5d553d438de2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.036196    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.136929    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.145644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.141065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.146615 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.146431    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:55.319102 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.237295    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.527575 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.337587    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.527575 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.438047    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.527575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.498009    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.538855    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.572964    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.573065    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.573091    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.637805    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:09:54 +0000 UTC" deadline="2028-01-23 02:18:34.891446746 +0000 UTC"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.637829    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13669h3m39.2536206s"
Jul 02 13:14:55.672981 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.640853    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.844879 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.741686    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:55.844879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:55.770576    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cd314639f46c104191661c02cd9b02.slice/crio-69566cee4d52eaebeb9b1aeb7ea787bc40dd31fab00b047ef89f7d7cb7e5c56b WatchSource:0}: Error finding container 69566cee4d52eaebeb9b1aeb7ea787bc40dd31fab00b047ef89f7d7cb7e5c56b: Status 404 returned error can't find the container with id 69566cee4d52eaebeb9b1aeb7ea787bc40dd31fab00b047ef89f7d7cb7e5c56b
Jul 02 13:14:55.844879 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:55.771320    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543120c32551afd3c62e5d553d438de2.slice/crio-81126bdaff21f6d0107d91eff6e8dd17c98c8923b1f591aef3e795cc90653a08 WatchSource:0}: Error finding container 81126bdaff21f6d0107d91eff6e8dd17c98c8923b1f591aef3e795cc90653a08: Status 404 returned error can't find the container with id 81126bdaff21f6d0107d91eff6e8dd17c98c8923b1f591aef3e795cc90653a08
Jul 02 13:14:55.844879 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.776808    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:14:55.844879 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:55.842265    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-102.ec2.internal\" not found"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.868975    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.927979    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.939179    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.940111    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.944803    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:56.040468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:55.946231    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:56.612644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.612572    2565 apiserver.go:52] "Watching apiserver"
Jul 02 13:14:56.623556 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.623533    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:14:56.624028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.623995    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w","openshift-image-registry/node-ca-rv7sq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal","openshift-network-operator/iptables-alerter-t5nw2","openshift-ovn-kubernetes/ovnkube-node-mprzv","kube-system/global-pull-secret-syncer-5d8ms","kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dhsvd","openshift-dns/node-resolver-v295b","openshift-multus/multus-9tszt","openshift-multus/multus-additional-cni-plugins-j4dp4","openshift-multus/network-metrics-daemon-ffcp9","openshift-network-diagnostics/network-check-target-55ttw","kube-system/konnectivity-agent-4scd4"]
Jul 02 13:14:56.626472 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.626447    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.627592 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.627573    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:56.628721 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.628703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:56.628828 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.628786    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:14:56.628926 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.628909    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:14:56.629019 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.628986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:14:56.629019 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.629007    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.629136 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.629078    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.629136 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.629085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-prjn6\""
Jul 02 13:14:56.630903 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.630848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.633148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.632994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:14:56.633148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.633029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pptw4\""
Jul 02 13:14:56.633148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.633107    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:14:56.633339 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.633298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.635568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.635550    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wrdzw\""
Jul 02 13:14:56.635630 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.635569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.635822 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.635806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.635902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.635809    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:14:56.636720 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.636703    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.637813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.637795    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.637904 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.637856    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:14:56.639249 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cnibin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639249 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639233    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:14:56.639406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-conf-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-multus-certs\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cni-binary-copy\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-bin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-multus\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-hostroot\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-etc-kubernetes\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv2w\" (UniqueName: \"kubernetes.io/projected/0a94fc30-e47f-488d-9c59-8e7fc264baa9-kube-api-access-4gv2w\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4hbxl\""
Jul 02 13:14:56.639608 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639526    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:14:56.639847 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-os-release\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.639917 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.639888    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.640372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.640356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.640690 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.640669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-socket-dir-parent\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.640813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.640795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-netns\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.640892 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.640875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-daemon-config\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.641041 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:56.641100 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-system-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.641353 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-k8s-cni-cncf-io\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.641405 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-kubelet\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.641688 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:14:56.641734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.641711    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.643111 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.643092    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:14:56.643187 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.643129    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-krghf\""
Jul 02 13:14:56.643370 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.643346    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.643694 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.643675    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:56.643774 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.643684    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.644082 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.644027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.647088 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.647014    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ms5t4\""
Jul 02 13:14:56.647179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.647110    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.647372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.647357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.647449 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.647385    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.649439 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.649424    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.649641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.649623    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:14:56.649738 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.649678    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p9fwg\""
Jul 02 13:14:56.649898 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.649881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.650014 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.649922    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.652057 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.652041    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:14:56.652194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.652176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:14:56.652298 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.652225    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:14:56.652298 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.652246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zwn2d\""
Jul 02 13:14:56.653005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.652963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.655311 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.655295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.655415 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.655358    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:14:56.655514 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.655496    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:14:56.655599 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.655513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:14:56.655665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.655648    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4882l\""
Jul 02 13:14:56.670046 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.670028    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tf82v"
Jul 02 13:14:56.679116 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.679100    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:56.680276 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.680258    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tf82v"
Jul 02 13:14:56.708068 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.708032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal" event={"ID":"67cd314639f46c104191661c02cd9b02","Type":"ContainerStarted","Data":"69566cee4d52eaebeb9b1aeb7ea787bc40dd31fab00b047ef89f7d7cb7e5c56b"}
Jul 02 13:14:56.709033 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.709015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal" event={"ID":"543120c32551afd3c62e5d553d438de2","Type":"ContainerStarted","Data":"81126bdaff21f6d0107d91eff6e8dd17c98c8923b1f591aef3e795cc90653a08"}
Jul 02 13:14:56.728632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.728611    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:14:56.742361 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-systemd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742453 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-netd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742453 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.742453 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742453 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-netns\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-daemon-config\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttlv\" (UniqueName: \"kubernetes.io/projected/91893a1e-ad7e-45b5-a53c-17c189ce5221-kube-api-access-pttlv\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742578    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-netns\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-bin\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-kubernetes\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-systemd\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-k8s-cni-cncf-io\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9039d09-c0b2-443f-bf7f-0c77b104026b-iptables-alerter-script\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742684    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-script-lib\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4mr\" (UniqueName: \"kubernetes.io/projected/6f926e87-2b39-40ae-9798-072eba46899d-kube-api-access-fp4mr\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742748    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-k8s-cni-cncf-io\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-registration-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-sys-fs\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-netns\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.742927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-system-cni-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cni-binary-copy\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.742990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-bin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-hostroot\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsr7d\" (UniqueName: \"kubernetes.io/projected/e9039d09-c0b2-443f-bf7f-0c77b104026b-kube-api-access-dsr7d\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743068    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-bin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743129    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-daemon-config\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-lib-modules\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743156    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-hostroot\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-serviceca\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9039d09-c0b2-443f-bf7f-0c77b104026b-host-slash\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-node-log\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-log-socket\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.743444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-run\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-sys\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743403    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cni-binary-copy\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-multus-certs\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-socket-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743509    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-run-multus-certs\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-kubelet\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-var-lib-kubelet\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-tmp\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-etc-kubernetes\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv2w\" (UniqueName: \"kubernetes.io/projected/0a94fc30-e47f-488d-9c59-8e7fc264baa9-kube-api-access-4gv2w\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-host\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743664    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-etc-kubernetes\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-device-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-conf\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-os-release\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-dbus\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743802    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-os-release\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91893a1e-ad7e-45b5-a53c-17c189ce5221-tmp-dir\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbw4\" (UniqueName: \"kubernetes.io/projected/ac070bdf-8694-478b-86dd-21ceb59c9858-kube-api-access-lwbw4\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-ovn\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-modprobe-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.743997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-etc-tuned\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-system-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-kubelet\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-kubelet-config\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-etc-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744118    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-system-cni-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af575503-ae25-49d2-8280-39c4d2d6fe16-ovn-node-metrics-cert\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744137    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-kubelet\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzpn\" (UniqueName: \"kubernetes.io/projected/206cd077-8764-42f9-86ca-71d3cc81fa57-kube-api-access-hrzpn\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.744848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cnibin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef581d1d-f306-4d3a-b07f-68a2fb289e78-konnectivity-ca\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462rd\" (UniqueName: \"kubernetes.io/projected/af575503-ae25-49d2-8280-39c4d2d6fe16-kube-api-access-462rd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhzf\" (UniqueName: \"kubernetes.io/projected/45afd23b-217f-47d3-85bc-0ae6e6390ece-kube-api-access-wbhzf\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-multus\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744333    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-var-lib-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744351    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-config\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744324    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-cnibin\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-os-release\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744414    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-host-var-lib-cni-multus\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-socket-dir-parent\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91893a1e-ad7e-45b5-a53c-17c189ce5221-hosts-file\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-socket-dir-parent\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysconfig\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-cnibin\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.745442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9968k\" (UniqueName: \"kubernetes.io/projected/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-kube-api-access-9968k\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-systemd-units\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-env-overrides\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-host\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.744981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-conf-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.745043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef581d1d-f306-4d3a-b07f-68a2fb289e78-agent-certs\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.745069    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a94fc30-e47f-488d-9c59-8e7fc264baa9-multus-conf-dir\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.746024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.745095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-slash\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.753078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.753047    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:14:56.753171 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.753132    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:56.753171 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.753157    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:56.753171 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.753171    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:56.753330 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.753256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:57.253217337 +0000 UTC m=+2.923024683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:56.757197 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.757176    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv2w\" (UniqueName: \"kubernetes.io/projected/0a94fc30-e47f-488d-9c59-8e7fc264baa9-kube-api-access-4gv2w\") pod \"multus-9tszt\" (UID: \"0a94fc30-e47f-488d-9c59-8e7fc264baa9\") " pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.845425 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-var-lib-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-config\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-os-release\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845527    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-var-lib-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91893a1e-ad7e-45b5-a53c-17c189ce5221-hosts-file\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.845596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysconfig\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-cnibin\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845623    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-os-release\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9968k\" (UniqueName: \"kubernetes.io/projected/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-kube-api-access-9968k\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-systemd-units\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845685    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysconfig\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-env-overrides\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-host\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845741    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91893a1e-ad7e-45b5-a53c-17c189ce5221-hosts-file\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef581d1d-f306-4d3a-b07f-68a2fb289e78-agent-certs\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845769    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-cnibin\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-slash\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-systemd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-netd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.845912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttlv\" (UniqueName: \"kubernetes.io/projected/91893a1e-ad7e-45b5-a53c-17c189ce5221-kube-api-access-pttlv\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.845974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-bin\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-kubernetes\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-systemd\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9039d09-c0b2-443f-bf7f-0c77b104026b-iptables-alerter-script\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-script-lib\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4mr\" (UniqueName: \"kubernetes.io/projected/6f926e87-2b39-40ae-9798-072eba46899d-kube-api-access-fp4mr\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-registration-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846158    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-config\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-sys-fs\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-netns\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846208    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-system-cni-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846255    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.846267    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:56.846749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846273    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-system-cni-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846304    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-systemd-units\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846337    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-slash\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsr7d\" (UniqueName: \"kubernetes.io/projected/e9039d09-c0b2-443f-bf7f-0c77b104026b-kube-api-access-dsr7d\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846364    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846412    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-registration-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846398    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-systemd\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846444    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-sys-fs\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846446    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-systemd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846494    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-netns\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-netd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846690    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-cni-bin\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.846762    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:57.346744116 +0000 UTC m=+3.016551466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846763    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-env-overrides\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846831    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-kubernetes\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.847641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846830    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e9039d09-c0b2-443f-bf7f-0c77b104026b-iptables-alerter-script\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846865    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-run-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846887    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-host\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.846984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-lib-modules\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847064    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af575503-ae25-49d2-8280-39c4d2d6fe16-ovnkube-script-lib\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847073    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45afd23b-217f-47d3-85bc-0ae6e6390ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-serviceca\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9039d09-c0b2-443f-bf7f-0c77b104026b-host-slash\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847238    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-node-log\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847282    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45afd23b-217f-47d3-85bc-0ae6e6390ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-log-socket\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.847309    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847322    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-log-socket\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.848432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847356    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847320    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9039d09-c0b2-443f-bf7f-0c77b104026b-host-slash\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:56.847372    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:57.347357463 +0000 UTC m=+3.017164814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847412    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-node-log\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847413    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-lib-modules\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-run\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-sys\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847468    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-run\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847496    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-sys\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-socket-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-kubelet\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847570    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-host-kubelet\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847569    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-var-lib-kubelet\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847594    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-socket-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.849466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-tmp\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847658    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-var-lib-kubelet\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-host\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847676    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-serviceca\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-device-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847706    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-host\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-conf\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-dbus\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91893a1e-ad7e-45b5-a53c-17c189ce5221-tmp-dir\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847778    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac070bdf-8694-478b-86dd-21ceb59c9858-device-dir\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbw4\" (UniqueName: \"kubernetes.io/projected/ac070bdf-8694-478b-86dd-21ceb59c9858-kube-api-access-lwbw4\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-ovn\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-modprobe-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-etc-tuned\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-kubelet-config\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847901    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-dbus\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847912    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-sysctl-conf\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-etc-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.850320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af575503-ae25-49d2-8280-39c4d2d6fe16-ovn-node-metrics-cert\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.847986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzpn\" (UniqueName: \"kubernetes.io/projected/206cd077-8764-42f9-86ca-71d3cc81fa57-kube-api-access-hrzpn\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef581d1d-f306-4d3a-b07f-68a2fb289e78-konnectivity-ca\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462rd\" (UniqueName: \"kubernetes.io/projected/af575503-ae25-49d2-8280-39c4d2d6fe16-kube-api-access-462rd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhzf\" (UniqueName: \"kubernetes.io/projected/45afd23b-217f-47d3-85bc-0ae6e6390ece-kube-api-access-wbhzf\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848107    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-run-ovn\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848205    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f926e87-2b39-40ae-9798-072eba46899d-etc-modprobe-d\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848228    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91893a1e-ad7e-45b5-a53c-17c189ce5221-tmp-dir\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848278    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38b4b3cd-994a-47be-a765-884a6af449c8-kubelet-config\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.848318    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af575503-ae25-49d2-8280-39c4d2d6fe16-etc-openvswitch\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.849034    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef581d1d-f306-4d3a-b07f-68a2fb289e78-konnectivity-ca\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.849995    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef581d1d-f306-4d3a-b07f-68a2fb289e78-agent-certs\") pod \"konnectivity-agent-4scd4\" (UID: \"ef581d1d-f306-4d3a-b07f-68a2fb289e78\") " pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.850212    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-tmp\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.850617    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f926e87-2b39-40ae-9798-072eba46899d-etc-tuned\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.851067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.850864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af575503-ae25-49d2-8280-39c4d2d6fe16-ovn-node-metrics-cert\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.858732 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.858685    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttlv\" (UniqueName: \"kubernetes.io/projected/91893a1e-ad7e-45b5-a53c-17c189ce5221-kube-api-access-pttlv\") pod \"node-resolver-v295b\" (UID: \"91893a1e-ad7e-45b5-a53c-17c189ce5221\") " pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.859388 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.859359    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4mr\" (UniqueName: \"kubernetes.io/projected/6f926e87-2b39-40ae-9798-072eba46899d-kube-api-access-fp4mr\") pod \"tuned-dhsvd\" (UID: \"6f926e87-2b39-40ae-9798-072eba46899d\") " pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.859791 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.859772    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9968k\" (UniqueName: \"kubernetes.io/projected/8f0cc46a-0ad9-4952-bf16-b87d344e38ee-kube-api-access-9968k\") pod \"node-ca-rv7sq\" (UID: \"8f0cc46a-0ad9-4952-bf16-b87d344e38ee\") " pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.860392 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.860367    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-462rd\" (UniqueName: \"kubernetes.io/projected/af575503-ae25-49d2-8280-39c4d2d6fe16-kube-api-access-462rd\") pod \"ovnkube-node-mprzv\" (UID: \"af575503-ae25-49d2-8280-39c4d2d6fe16\") " pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.864420 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.864370    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsr7d\" (UniqueName: \"kubernetes.io/projected/e9039d09-c0b2-443f-bf7f-0c77b104026b-kube-api-access-dsr7d\") pod \"iptables-alerter-t5nw2\" (UID: \"e9039d09-c0b2-443f-bf7f-0c77b104026b\") " pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.866001 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.865981    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzpn\" (UniqueName: \"kubernetes.io/projected/206cd077-8764-42f9-86ca-71d3cc81fa57-kube-api-access-hrzpn\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:56.866575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.866559    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbw4\" (UniqueName: \"kubernetes.io/projected/ac070bdf-8694-478b-86dd-21ceb59c9858-kube-api-access-lwbw4\") pod \"aws-ebs-csi-driver-node-j4b7w\" (UID: \"ac070bdf-8694-478b-86dd-21ceb59c9858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.866636 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.866592    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhzf\" (UniqueName: \"kubernetes.io/projected/45afd23b-217f-47d3-85bc-0ae6e6390ece-kube-api-access-wbhzf\") pod \"multus-additional-cni-plugins-j4dp4\" (UID: \"45afd23b-217f-47d3-85bc-0ae6e6390ece\") " pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:56.937107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.937072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9tszt"
Jul 02 13:14:56.947604 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.947584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:14:56.957318 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.957300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t5nw2"
Jul 02 13:14:56.962442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.962422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:14:56.968993 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.968976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v295b"
Jul 02 13:14:56.975648 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.975620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dhsvd"
Jul 02 13:14:56.982182 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.982164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rv7sq"
Jul 02 13:14:56.989797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.989780    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w"
Jul 02 13:14:56.995458 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:56.995435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4dp4"
Jul 02 13:14:57.036821 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.036795    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef581d1d_f306_4d3a_b07f_68a2fb289e78.slice/crio-270adaf5cd8d678943134340485d1ce42b60fd178a08ee9a83a3c6b66355fd3b WatchSource:0}: Error finding container 270adaf5cd8d678943134340485d1ce42b60fd178a08ee9a83a3c6b66355fd3b: Status 404 returned error can't find the container with id 270adaf5cd8d678943134340485d1ce42b60fd178a08ee9a83a3c6b66355fd3b
Jul 02 13:14:57.039884 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.039860    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac070bdf_8694_478b_86dd_21ceb59c9858.slice/crio-3d235987bd36fe0746ca4d33cc516d0369be42687298d4b4bbf08f238fb77589 WatchSource:0}: Error finding container 3d235987bd36fe0746ca4d33cc516d0369be42687298d4b4bbf08f238fb77589: Status 404 returned error can't find the container with id 3d235987bd36fe0746ca4d33cc516d0369be42687298d4b4bbf08f238fb77589
Jul 02 13:14:57.041226 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.041103    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9039d09_c0b2_443f_bf7f_0c77b104026b.slice/crio-c4085ca9018ca21b164c13c6e45ddc8dac041b37c63a1507b0fa394ace625708 WatchSource:0}: Error finding container c4085ca9018ca21b164c13c6e45ddc8dac041b37c63a1507b0fa394ace625708: Status 404 returned error can't find the container with id c4085ca9018ca21b164c13c6e45ddc8dac041b37c63a1507b0fa394ace625708
Jul 02 13:14:57.041910 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.041884    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf575503_ae25_49d2_8280_39c4d2d6fe16.slice/crio-7e566a58329a660a06f113e5a1f4c5dbf811f1ea058ee2f07f431d39cbc75443 WatchSource:0}: Error finding container 7e566a58329a660a06f113e5a1f4c5dbf811f1ea058ee2f07f431d39cbc75443: Status 404 returned error can't find the container with id 7e566a58329a660a06f113e5a1f4c5dbf811f1ea058ee2f07f431d39cbc75443
Jul 02 13:14:57.042846 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.042769    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45afd23b_217f_47d3_85bc_0ae6e6390ece.slice/crio-d198f32fac2db71576158f0fffdb83eb99bcc6af85718c00cba65918e3f77310 WatchSource:0}: Error finding container d198f32fac2db71576158f0fffdb83eb99bcc6af85718c00cba65918e3f77310: Status 404 returned error can't find the container with id d198f32fac2db71576158f0fffdb83eb99bcc6af85718c00cba65918e3f77310
Jul 02 13:14:57.044392 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.044330    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a94fc30_e47f_488d_9c59_8e7fc264baa9.slice/crio-51de4d60420dfbe544434d3f94ce2695d344d98e60bbea24d14e2b319b8babbb WatchSource:0}: Error finding container 51de4d60420dfbe544434d3f94ce2695d344d98e60bbea24d14e2b319b8babbb: Status 404 returned error can't find the container with id 51de4d60420dfbe544434d3f94ce2695d344d98e60bbea24d14e2b319b8babbb
Jul 02 13:14:57.045500 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.045480    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91893a1e_ad7e_45b5_a53c_17c189ce5221.slice/crio-e6c9110380a5d502dc3e32461f347a7552cbd905c6bc3e2144bb5ad16e86f176 WatchSource:0}: Error finding container e6c9110380a5d502dc3e32461f347a7552cbd905c6bc3e2144bb5ad16e86f176: Status 404 returned error can't find the container with id e6c9110380a5d502dc3e32461f347a7552cbd905c6bc3e2144bb5ad16e86f176
Jul 02 13:14:57.066803 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.066778    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0cc46a_0ad9_4952_bf16_b87d344e38ee.slice/crio-f0127f52cc911772316608a7d16c27691fe5716bc646a94bbf08eea39d01e9b4 WatchSource:0}: Error finding container f0127f52cc911772316608a7d16c27691fe5716bc646a94bbf08eea39d01e9b4: Status 404 returned error can't find the container with id f0127f52cc911772316608a7d16c27691fe5716bc646a94bbf08eea39d01e9b4
Jul 02 13:14:57.067806 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:14:57.067782    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f926e87_2b39_40ae_9798_072eba46899d.slice/crio-223d238127523b538671530aae326c15f58d024903fc4fddd8e09b420cdb4856 WatchSource:0}: Error finding container 223d238127523b538671530aae326c15f58d024903fc4fddd8e09b420cdb4856: Status 404 returned error can't find the container with id 223d238127523b538671530aae326c15f58d024903fc4fddd8e09b420cdb4856
Jul 02 13:14:57.352451 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.352255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:57.352568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.352482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:57.352568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.352527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:57.352688 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352398    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352728    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.352709981 +0000 UTC m=+4.022517330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352746    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352785    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.352772849 +0000 UTC m=+4.022580208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352668    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352811    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:57.352835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352824    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:57.353149 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:57.352856    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.352846286 +0000 UTC m=+4.022653648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:57.683333 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.683163    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:56 +0000 UTC" deadline="2028-02-03 23:21:03.293168494 +0000 UTC"
Jul 02 13:14:57.683333 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.683217    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13954h6m5.60995512s"
Jul 02 13:14:57.722065 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.722025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rv7sq" event={"ID":"8f0cc46a-0ad9-4952-bf16-b87d344e38ee","Type":"ContainerStarted","Data":"f0127f52cc911772316608a7d16c27691fe5716bc646a94bbf08eea39d01e9b4"}
Jul 02 13:14:57.729357 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.729177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9tszt" event={"ID":"0a94fc30-e47f-488d-9c59-8e7fc264baa9","Type":"ContainerStarted","Data":"51de4d60420dfbe544434d3f94ce2695d344d98e60bbea24d14e2b319b8babbb"}
Jul 02 13:14:57.733301 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.733245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerStarted","Data":"d198f32fac2db71576158f0fffdb83eb99bcc6af85718c00cba65918e3f77310"}
Jul 02 13:14:57.739886 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.739824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w" event={"ID":"ac070bdf-8694-478b-86dd-21ceb59c9858","Type":"ContainerStarted","Data":"3d235987bd36fe0746ca4d33cc516d0369be42687298d4b4bbf08f238fb77589"}
Jul 02 13:14:57.743565 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.743502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4scd4" event={"ID":"ef581d1d-f306-4d3a-b07f-68a2fb289e78","Type":"ContainerStarted","Data":"270adaf5cd8d678943134340485d1ce42b60fd178a08ee9a83a3c6b66355fd3b"}
Jul 02 13:14:57.754644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.754612    2565 generic.go:356] "Generic (PLEG): container finished" podID="543120c32551afd3c62e5d553d438de2" containerID="66bddfe91b3ce13643af0663fb0b5e0f0ab4ed1f9cc654c8e423e8ba3db36e7a" exitCode=0
Jul 02 13:14:57.754745 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.754693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal" event={"ID":"543120c32551afd3c62e5d553d438de2","Type":"ContainerDied","Data":"66bddfe91b3ce13643af0663fb0b5e0f0ab4ed1f9cc654c8e423e8ba3db36e7a"}
Jul 02 13:14:57.758149 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.758100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dhsvd" event={"ID":"6f926e87-2b39-40ae-9798-072eba46899d","Type":"ContainerStarted","Data":"223d238127523b538671530aae326c15f58d024903fc4fddd8e09b420cdb4856"}
Jul 02 13:14:57.760852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.760797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v295b" event={"ID":"91893a1e-ad7e-45b5-a53c-17c189ce5221","Type":"ContainerStarted","Data":"e6c9110380a5d502dc3e32461f347a7552cbd905c6bc3e2144bb5ad16e86f176"}
Jul 02 13:14:57.762714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.762623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"7e566a58329a660a06f113e5a1f4c5dbf811f1ea058ee2f07f431d39cbc75443"}
Jul 02 13:14:57.765502 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:57.765448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t5nw2" event={"ID":"e9039d09-c0b2-443f-bf7f-0c77b104026b","Type":"ContainerStarted","Data":"c4085ca9018ca21b164c13c6e45ddc8dac041b37c63a1507b0fa394ace625708"}
Jul 02 13:14:58.303975 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.303929    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:58.361526 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.361487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:58.361703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.361551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:58.361703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.361602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:58.361810 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.361723    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.361810 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.361785    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.361768427 +0000 UTC m=+6.031575780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.362196 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362177    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.362262 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362233    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.362217822 +0000 UTC m=+6.032025168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.362364 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362310    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:58.362364 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362327    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:58.362364 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362339    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.362514 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.362372    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:00.362361248 +0000 UTC m=+6.032168596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.683436 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.683341    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:56 +0000 UTC" deadline="2027-12-04 19:01:10.178739889 +0000 UTC"
Jul 02 13:14:58.683436 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.683384    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12485h46m11.49536689s"
Jul 02 13:14:58.703237 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.703195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:14:58.703372 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.703324    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:14:58.703862 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.703820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:14:58.703979 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.703919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:14:58.704123 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:14:58.704107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:14:58.704226 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:14:58.704207    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:00.378566 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.378524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.378590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.378631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378769    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378789    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378801    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378852    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378864    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.378845684 +0000 UTC m=+10.048653050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378886    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.378876171 +0000 UTC m=+10.048683517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378804    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:00.379089 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.378921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:04.378912684 +0000 UTC m=+10.048720045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.702340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.702373    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:00.702340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.702468    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.702569    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:00.703008 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:00.702647    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:02.701916 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:02.701886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:02.702344 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:02.702039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:02.702635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:02.702616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:02.702737 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:02.702715    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:02.702795 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:02.702723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:02.702844 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:02.702804    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:04.411380 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.411330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:04.411736 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.411436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:04.411736 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.411486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:04.411736 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411663    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:04.411884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411764    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:12.411741252 +0000 UTC m=+18.081548600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:04.411884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411797    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:04.411884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411859    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:12.411840868 +0000 UTC m=+18.081648214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:04.412066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411963    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:04.412066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411987    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:04.412066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.411999    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:04.412066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.412047    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:12.412032864 +0000 UTC m=+18.081840216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.702863    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.702959    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.702997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.703104    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:04.703152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:04.704328 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:04.703238    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:06.703181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:06.703143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:06.703630 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:06.703143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:06.703630 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:06.703276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:06.703630 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:06.703279    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:06.703630 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:06.703349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:06.703630 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:06.703440    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:08.701596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:08.701555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:08.702084 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:08.701602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:08.702084 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:08.701676    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:08.702084 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:08.701755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:08.702084 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:08.701854    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:08.702084 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:08.701918    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:10.702487 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:10.702449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:10.702962 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:10.702452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:10.702962 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:10.702587    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:10.702962 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:10.702646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:10.702962 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:10.702458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:10.702962 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:10.702727    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:12.474093 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.474057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.474111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.474151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474207    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474256    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474273    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474273    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474286    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474275    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:28.474255891 +0000 UTC m=+34.144063256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474327    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:28.474316387 +0000 UTC m=+34.144123746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:12.474581 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.474340    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:28.474332938 +0000 UTC m=+34.144140284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:12.704988 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.704918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:12.705137 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.705036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:12.705137 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.705049    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:12.705137 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:12.705068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:12.705286 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.705186    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:12.705286 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:12.705276    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:14.703094 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:14.703044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:14.703533 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:14.703208    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:14.703533 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:14.703219    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:14.703533 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:14.703326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:14.703533 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:14.703345    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:14.703533 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:14.703460    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:16.702256 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.701804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:16.702906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.701800    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:16.702906 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:16.702376    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:16.702906 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:16.702438    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:16.702906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.701800    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:16.702906 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:16.702533    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:16.807776 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.807745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v295b" event={"ID":"91893a1e-ad7e-45b5-a53c-17c189ce5221","Type":"ContainerStarted","Data":"8d6a409b179b630c3a02457ca6128ccaaebb452864c3ee82e85c75518fc2d2fc"}
Jul 02 13:15:16.812712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"f317c53d1106605e26b02acd6e9310efa8d6619a753062d2483e737d60d6335c"}
Jul 02 13:15:16.812852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"6e8e37ba0987a9c5a31814a1202f035251233b0bb55a5f5dfdece8600fe70826"}
Jul 02 13:15:16.812852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"ca190a39b5a51d27214d89a1a71f7cf98db5a02803a239ad8760c6a7c477f2a9"}
Jul 02 13:15:16.812852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"4ea9df5028a850fe655760bf7a9932d98aca3a653d8bb4a9481a44d882e09366"}
Jul 02 13:15:16.813035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"d0df5c6819cbd13a176eb26834429ad6c652b9b3abe46d8e13ad28c362ad0062"}
Jul 02 13:15:16.813035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.812966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"809812cf392a217abdbfb930b18ff280a12227765290a05435172e17c6c8a538"}
Jul 02 13:15:16.814529 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.814501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal" event={"ID":"67cd314639f46c104191661c02cd9b02","Type":"ContainerStarted","Data":"19cd7bc660ce6d1752f7b38d138478e618b038e8d122c644394f6eb81a786fea"}
Jul 02 13:15:16.815985 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.815963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rv7sq" event={"ID":"8f0cc46a-0ad9-4952-bf16-b87d344e38ee","Type":"ContainerStarted","Data":"540f157391c6708bc95d6c256f16fa6a4cc2288f992be46c1baabeba6d652e0e"}
Jul 02 13:15:16.817384 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.817364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9tszt" event={"ID":"0a94fc30-e47f-488d-9c59-8e7fc264baa9","Type":"ContainerStarted","Data":"c5e18987dcf607faa63bdfe9ed970aca3a0ce5a82b534eeb9cc94c9fc4799424"}
Jul 02 13:15:16.818895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.818862    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="d90cc3cb909ac3e729c88f542d525aab0e4f24e00c7c9081f0c4a321b2d0ca0e" exitCode=0
Jul 02 13:15:16.819005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.818961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"d90cc3cb909ac3e729c88f542d525aab0e4f24e00c7c9081f0c4a321b2d0ca0e"}
Jul 02 13:15:16.820591 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.820568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w" event={"ID":"ac070bdf-8694-478b-86dd-21ceb59c9858","Type":"ContainerStarted","Data":"b00e099b46d89366bb2caafc2d241e58da4d708962da58ab9997fa8e2081495b"}
Jul 02 13:15:16.822193 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.822099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4scd4" event={"ID":"ef581d1d-f306-4d3a-b07f-68a2fb289e78","Type":"ContainerStarted","Data":"fea0d5452acb3ea88d9a2e05f2c07a5deb89faffacc19d83b013b34047032ba2"}
Jul 02 13:15:16.823973 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.823955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal" event={"ID":"543120c32551afd3c62e5d553d438de2","Type":"ContainerStarted","Data":"98d9262734306707a07955f2a8f73dad4202d4e57c574173d47cc1966000798a"}
Jul 02 13:15:16.825431 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.825402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dhsvd" event={"ID":"6f926e87-2b39-40ae-9798-072eba46899d","Type":"ContainerStarted","Data":"3f38aab34c13de59bfece750935463ccebf3af8260e2cca354b57d67dd26f37e"}
Jul 02 13:15:16.826855 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.826818    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v295b" podStartSLOduration=3.944780968 podStartE2EDuration="22.826806459s" totalImagesPullingTime="18.882025491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.826539799 +0000 UTC m=+22.496347167" watchObservedRunningTime="2026-07-02 13:15:16.826806459 +0000 UTC m=+22.496613826"
Jul 02 13:15:16.849718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.849678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dhsvd" podStartSLOduration=3.971182145 podStartE2EDuration="22.849665792s" totalImagesPullingTime="18.878483647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.848747523 +0000 UTC m=+22.518554893" watchObservedRunningTime="2026-07-02 13:15:16.849665792 +0000 UTC m=+22.519473160"
Jul 02 13:15:16.866244 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.866201    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rv7sq" podStartSLOduration=4.059472503 podStartE2EDuration="22.866183706s" totalImagesPullingTime="18.806711203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.866043894 +0000 UTC m=+22.535851262" watchObservedRunningTime="2026-07-02 13:15:16.866183706 +0000 UTC m=+22.535991075"
Jul 02 13:15:16.883233 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.883192    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4scd4" podStartSLOduration=4.044881731 podStartE2EDuration="22.88317891s" totalImagesPullingTime="18.838297179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.88283301 +0000 UTC m=+22.552640378" watchObservedRunningTime="2026-07-02 13:15:16.88317891 +0000 UTC m=+22.552986277"
Jul 02 13:15:16.943296 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.943243    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-102.ec2.internal" podStartSLOduration=21.943228257 podStartE2EDuration="21.943228257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.943209028 +0000 UTC m=+22.613016397" watchObservedRunningTime="2026-07-02 13:15:16.943228257 +0000 UTC m=+22.613035630"
Jul 02 13:15:16.943665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.943644    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-102.ec2.internal" podStartSLOduration=21.943636481 podStartE2EDuration="21.943636481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.926280388 +0000 UTC m=+22.596087756" watchObservedRunningTime="2026-07-02 13:15:16.943636481 +0000 UTC m=+22.613443849"
Jul 02 13:15:16.960834 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:16.960792    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9tszt" podStartSLOduration=3.888286154 podStartE2EDuration="22.960781278s" totalImagesPullingTime="19.072495124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.960419808 +0000 UTC m=+22.630227191" watchObservedRunningTime="2026-07-02 13:15:16.960781278 +0000 UTC m=+22.630588646"
Jul 02 13:15:17.032511 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.032482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:15:17.033005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.032987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:15:17.441084 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.441065    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:15:17.713722 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.713563    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:15:17.441079217Z","UUID":"8e08ecb4-65e8-4e2e-823f-d8fbeb4411cd","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:15:17.716776 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.716750    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:15:17.716913 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.716796    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:15:17.830073 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.830036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t5nw2" event={"ID":"e9039d09-c0b2-443f-bf7f-0c77b104026b","Type":"ContainerStarted","Data":"bccbd7762ee0de2d5891f8bff050ebe1e93277f9d147d67a372013d6bbb462e3"}
Jul 02 13:15:17.832237 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.832202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w" event={"ID":"ac070bdf-8694-478b-86dd-21ceb59c9858","Type":"ContainerStarted","Data":"a24bc46836acacb3a98ef04418b92c283123a43ff2765690132e6bf7ce5bd39e"}
Jul 02 13:15:17.832910 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.832882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:15:17.833291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.833274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4scd4"
Jul 02 13:15:17.855236 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:17.855196    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t5nw2" podStartSLOduration=4.948362654 podStartE2EDuration="23.855183788s" totalImagesPullingTime="18.906821134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:17.85441067 +0000 UTC m=+23.524218040" watchObservedRunningTime="2026-07-02 13:15:17.855183788 +0000 UTC m=+23.524991155"
Jul 02 13:15:18.701570 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.701542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:18.701746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.701543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:18.701746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.701551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:18.701746 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:18.701680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:18.701908 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:18.701801    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:18.701988 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:18.701908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:18.838819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.838536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"9a0a2764084b87061a8af7cac30244ca6391239db4a0d9cf80fe1de663df083f"}
Jul 02 13:15:18.841379 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.841350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w" event={"ID":"ac070bdf-8694-478b-86dd-21ceb59c9858","Type":"ContainerStarted","Data":"2b870cd7a0189d7a91646e2190a789d2a761711e4fb8515a4a891d122c0b5170"}
Jul 02 13:15:18.861235 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:18.861192    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4b7w" podStartSLOduration=3.730242087 podStartE2EDuration="24.861173079s" totalImagesPullingTime="21.130930992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:18.860777874 +0000 UTC m=+24.530585243" watchObservedRunningTime="2026-07-02 13:15:18.861173079 +0000 UTC m=+24.530980451"
Jul 02 13:15:20.702143 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:20.702113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:20.702797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:20.702113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:20.702797 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:20.702214    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:20.702797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:20.702113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:20.702797 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:20.702327    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:20.702797 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:20.702382    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:21.849860 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.849672    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="eecb632bc3a2d4ae176fda5339bb1e2fbc132825bd0da7a5af7fcc71bf877ab2" exitCode=0
Jul 02 13:15:21.850599 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.849754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"eecb632bc3a2d4ae176fda5339bb1e2fbc132825bd0da7a5af7fcc71bf877ab2"}
Jul 02 13:15:21.853473 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.853451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" event={"ID":"af575503-ae25-49d2-8280-39c4d2d6fe16","Type":"ContainerStarted","Data":"c1c348b31efaedd3604b0382a03cb50a66b60c6b69f3e03aade771a66cec69a1"}
Jul 02 13:15:21.853791 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.853776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:21.853871 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.853843    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:21.868740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.868716    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:21.868873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.868809    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:21.901111 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:21.901060    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv" podStartSLOduration=8.942008137 podStartE2EDuration="27.901044204s" totalImagesPullingTime="18.959036067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:21.899343464 +0000 UTC m=+27.569150835" watchObservedRunningTime="2026-07-02 13:15:21.901044204 +0000 UTC m=+27.570851571"
Jul 02 13:15:22.167553 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:22.167452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:22.701821 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:22.701784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:22.701821 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:22.701814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:22.702070 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:22.701784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:22.702070 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:22.701918    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:22.702070 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:22.702002    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:22.702070 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:22.702063    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:23.492049 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.491830    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5d8ms"]
Jul 02 13:15:23.492348 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.492124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:23.492348 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:23.492207    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:23.494980 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.494929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55ttw"]
Jul 02 13:15:23.495094 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.495054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:23.495596 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:23.495144    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:23.495873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.495733    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ffcp9"]
Jul 02 13:15:23.495873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.495821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:23.496017 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:23.495922    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:23.859973 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.859927    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="6b34de31866c1181137fddf35ee471c6368db9a55c0074e7f92c14dce979aca8" exitCode=0
Jul 02 13:15:23.860137 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:23.860016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"6b34de31866c1181137fddf35ee471c6368db9a55c0074e7f92c14dce979aca8"}
Jul 02 13:15:24.074131 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.074096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.163035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.162953    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.228439 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.228399    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.514465 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.514373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.661535 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.661498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.703225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.703188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:24.703453 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:24.703320    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:24.752185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.752146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.816754 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.816719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.871400 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:24.871372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.473918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.473886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.542091 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.542054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.701713 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.701677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:25.701884 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.701686    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:25.701884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:25.701800    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:25.701884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:25.701857    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:25.871978 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.871894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.929708 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:25.929673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:26.702173 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:26.702081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:26.702626 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:26.702228    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ffcp9" podUID="206cd077-8764-42f9-86ca-71d3cc81fa57"
Jul 02 13:15:26.869217 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:26.869180    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="a8f7c343ea34d3ea518f5614f7b8d6bf5e8f0786553f73a2147e513c78ebc5d6" exitCode=0
Jul 02 13:15:26.869365 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:26.869219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"a8f7c343ea34d3ea518f5614f7b8d6bf5e8f0786553f73a2147e513c78ebc5d6"}
Jul 02 13:15:27.702597 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:27.702459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:27.702597 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:27.702561    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55ttw" podUID="9f73406e-e073-47bb-b010-315f56ab60d8"
Jul 02 13:15:27.703209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:27.702623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:27.703209 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:27.702742    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d8ms" podUID="38b4b3cd-994a-47be-a765-884a6af449c8"
Jul 02 13:15:28.412563 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.412523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.468359 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.468322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.496161 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.496080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:28.496161 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.496152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.496190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496205    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496225    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496239    2565 projected.go:196] Error preparing data for projected volume kube-api-access-s4c65 for pod openshift-network-diagnostics/network-check-target-55ttw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496272    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496294    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65 podName:9f73406e-e073-47bb-b010-315f56ab60d8 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.496276031 +0000 UTC m=+66.166083396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4c65" (UniqueName: "kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65") pod "network-check-target-55ttw" (UID: "9f73406e-e073-47bb-b010-315f56ab60d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496310    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.496301818 +0000 UTC m=+66.166109166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:28.496389 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496343    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:28.496860 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:28.496415    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret podName:38b4b3cd-994a-47be-a765-884a6af449c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.496396312 +0000 UTC m=+66.166203663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret") pod "global-pull-secret-syncer-5d8ms" (UID: "38b4b3cd-994a-47be-a765-884a6af449c8") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:28.696644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.696571    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-102.ec2.internal" event="NodeReady"
Jul 02 13:15:28.696798 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.696709    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:15:28.702062 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.702034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:15:28.704426 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.704403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:28.704426 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.704411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qblmr\""
Jul 02 13:15:28.746437 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.746410    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4d98s"]
Jul 02 13:15:28.749841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.749813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:28.750201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.750179    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vm7c9"]
Jul 02 13:15:28.752314 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.752289    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrpc5\""
Jul 02 13:15:28.752418 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.752377    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:15:28.752418 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.752397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:15:28.753646 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.753628    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:28.755747 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.755731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:15:28.756023 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.756007    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:15:28.756104 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.756052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:15:28.756263 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.756248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-phqj7\""
Jul 02 13:15:28.760448 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.760397    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4d98s"]
Jul 02 13:15:28.762812 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.762754    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vm7c9"]
Jul 02 13:15:28.899336 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57480-bc69-4de0-954a-098c7f6e0903-config-volume\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:28.899336 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d57480-bc69-4de0-954a-098c7f6e0903-tmp-dir\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:28.899575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpvh\" (UniqueName: \"kubernetes.io/projected/c2d57480-bc69-4de0-954a-098c7f6e0903-kube-api-access-fcpvh\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:28.899575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-cert\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:28.899575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:28.899575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:28.899496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-kube-api-access-rswf2\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.000250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57480-bc69-4de0-954a-098c7f6e0903-config-volume\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.000250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d57480-bc69-4de0-954a-098c7f6e0903-tmp-dir\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.000250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpvh\" (UniqueName: \"kubernetes.io/projected/c2d57480-bc69-4de0-954a-098c7f6e0903-kube-api-access-fcpvh\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.000503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-cert\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.000503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.000503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-kube-api-access-rswf2\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.000652 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000594    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d57480-bc69-4de0-954a-098c7f6e0903-tmp-dir\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.000705 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:29.000695    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:29.000768 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:29.000753    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:29.500734419 +0000 UTC m=+35.170541805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:29.000905 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.000887    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57480-bc69-4de0-954a-098c7f6e0903-config-volume\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.005232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.005057    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-cert\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.013691 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.013670    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpvh\" (UniqueName: \"kubernetes.io/projected/c2d57480-bc69-4de0-954a-098c7f6e0903-kube-api-access-fcpvh\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.013849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.013817    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswf2\" (UniqueName: \"kubernetes.io/projected/b1311dc9-756f-41d6-8749-bfdbd03fe4cd-kube-api-access-rswf2\") pod \"ingress-canary-vm7c9\" (UID: \"b1311dc9-756f-41d6-8749-bfdbd03fe4cd\") " pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.066854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.066823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vm7c9"
Jul 02 13:15:29.214292 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.214260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vm7c9"]
Jul 02 13:15:29.217697 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:15:29.217668    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1311dc9_756f_41d6_8749_bfdbd03fe4cd.slice/crio-5d81a977c8f0d945e5a2b68b49073d0ca4df6766dc27cafbfe05020e06c83609 WatchSource:0}: Error finding container 5d81a977c8f0d945e5a2b68b49073d0ca4df6766dc27cafbfe05020e06c83609: Status 404 returned error can't find the container with id 5d81a977c8f0d945e5a2b68b49073d0ca4df6766dc27cafbfe05020e06c83609
Jul 02 13:15:29.504075 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.504041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:29.504236 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:29.504195    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:29.504277 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:29.504267    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:30.504248001 +0000 UTC m=+36.174055353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:29.572215 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.572165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.631178 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.631142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.701554 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.701524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:15:29.701554 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.701553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:15:29.705179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.705137    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ncrqc\""
Jul 02 13:15:29.706052 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.706028    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:29.706160 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.706101    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:29.706160 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.706029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:29.876875 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:29.876845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vm7c9" event={"ID":"b1311dc9-756f-41d6-8749-bfdbd03fe4cd","Type":"ContainerStarted","Data":"5d81a977c8f0d945e5a2b68b49073d0ca4df6766dc27cafbfe05020e06c83609"}
Jul 02 13:15:30.511647 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:30.511606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:30.511847 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:30.511763    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:30.511847 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:30.511836    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:32.511819115 +0000 UTC m=+38.181626461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:32.526903 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:32.526865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:32.527289 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:32.526986    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:32.527289 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:32.527038    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:36.52702505 +0000 UTC m=+42.196832401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:32.889113 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:32.888924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerStarted","Data":"3ed409240124bbefc7557f977cd3013e0ce8c3430ae882aedaec41cb1a0f8731"}
Jul 02 13:15:33.543132 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.543072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:33.603250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.603220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:33.893484 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.893411    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="3ed409240124bbefc7557f977cd3013e0ce8c3430ae882aedaec41cb1a0f8731" exitCode=0
Jul 02 13:15:33.893635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.893497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"3ed409240124bbefc7557f977cd3013e0ce8c3430ae882aedaec41cb1a0f8731"}
Jul 02 13:15:33.894947 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.894913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vm7c9" event={"ID":"b1311dc9-756f-41d6-8749-bfdbd03fe4cd","Type":"ContainerStarted","Data":"27eb43c428f227605848da4404ed5a0e51f279f1b3d0a75481f7e7ef99c0dc2e"}
Jul 02 13:15:33.935484 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:33.935387    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vm7c9" podStartSLOduration=2.402647477 podStartE2EDuration="5.935366344s" totalImagesPullingTime="3.532718867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:33.929416431 +0000 UTC m=+39.599223801" watchObservedRunningTime="2026-07-02 13:15:33.935366344 +0000 UTC m=+39.605173704"
Jul 02 13:15:34.899628 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:34.899593    2565 generic.go:356] "Generic (PLEG): container finished" podID="45afd23b-217f-47d3-85bc-0ae6e6390ece" containerID="48e90d274868f3c10cdaca4fc62e1c295eb591c4fe421bab86d9ed7c303295a6" exitCode=0
Jul 02 13:15:34.899986 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:34.899690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerDied","Data":"48e90d274868f3c10cdaca4fc62e1c295eb591c4fe421bab86d9ed7c303295a6"}
Jul 02 13:15:35.905465 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:35.905426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" event={"ID":"45afd23b-217f-47d3-85bc-0ae6e6390ece","Type":"ContainerStarted","Data":"7c146979dbff39b6e2089855606c1c42983a9e89df61928ef91027a64fe30dda"}
Jul 02 13:15:35.934125 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:35.934084    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j4dp4" podStartSLOduration=8.693123554 podStartE2EDuration="41.934070786s" totalImagesPullingTime="33.240947232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:35.9323101 +0000 UTC m=+41.602117462" watchObservedRunningTime="2026-07-02 13:15:35.934070786 +0000 UTC m=+41.603878132"
Jul 02 13:15:36.072288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:36.072263    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:36.136610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:36.136578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:36.554884 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:36.554845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:36.555044 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:36.555010    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:36.555086 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:36.555074    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:44.555059084 +0000 UTC m=+50.224866430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:44.608804 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:44.608764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:15:44.609275 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:44.608900    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:44.609275 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:44.608976    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.608958513 +0000 UTC m=+66.278765859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:45.911869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:45.911839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:46.358905 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:46.358870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:47.872916 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:47.872883    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:47.974352 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:47.974323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:48.863771 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:48.863736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:48.921448 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:48.921419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:49.583948 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:49.583900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:49.679889 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:49.679854    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.814322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:50.814277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:53.874405 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:53.874378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mprzv"
Jul 02 13:15:54.688662 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.688613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:54.760596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.760567    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"]
Jul 02 13:15:54.761058 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.761031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:54.775769 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.775747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.776550 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.776530    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"]
Jul 02 13:15:54.780300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.778596    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 13:15:54.782598 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.782575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ff6bd\""
Jul 02 13:15:54.783564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.783541    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 13:15:54.783701 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.783682    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:15:54.785005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.784986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:15:54.862384 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.862353    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58f7cb747c-qh9k6"]
Jul 02 13:15:54.872647 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.872626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.873082 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.873062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a639313a-e342-4393-a017-c9ee4dd9da02-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.873173 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.873117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5zk\" (UniqueName: \"kubernetes.io/projected/a639313a-e342-4393-a017-c9ee4dd9da02-kube-api-access-hx5zk\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.873173 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.873157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.874876 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.874861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 13:15:54.875248 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.874863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 13:15:54.875248 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.875159    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 13:15:54.875248 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.875181    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 13:15:54.875248 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.875240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wpnt9\""
Jul 02 13:15:54.875474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.875460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 13:15:54.875539 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.875525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 13:15:54.879574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.879556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58f7cb747c-qh9k6"]
Jul 02 13:15:54.973480 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a639313a-e342-4393-a017-c9ee4dd9da02-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.973480 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-default-certificate\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.973616 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.973616 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwj7\" (UniqueName: \"kubernetes.io/projected/e0f775f1-3efe-4432-b60e-f184557e7f60-kube-api-access-9zwj7\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.973712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5zk\" (UniqueName: \"kubernetes.io/projected/a639313a-e342-4393-a017-c9ee4dd9da02-kube-api-access-hx5zk\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.973750 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.973794 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.973845 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.973796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-stats-auth\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:54.973894 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:54.973848    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:54.973967 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:54.973911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:55.473893841 +0000 UTC m=+61.143701202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:54.974157 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.974140    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a639313a-e342-4393-a017-c9ee4dd9da02-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.984268 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.984244    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5zk\" (UniqueName: \"kubernetes.io/projected/a639313a-e342-4393-a017-c9ee4dd9da02-kube-api-access-hx5zk\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:54.994208 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:54.994187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:55.074379 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.074355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.074468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.074389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-stats-auth\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.074468 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.074430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-default-certificate\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.074542 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.074504    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:55.574489818 +0000 UTC m=+61.244297164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:55.074586 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.074541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.074633 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.074582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwj7\" (UniqueName: \"kubernetes.io/projected/e0f775f1-3efe-4432-b60e-f184557e7f60-kube-api-access-9zwj7\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.074769 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.074688    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:55.074769 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.074760    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:55.574744122 +0000 UTC m=+61.244551469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:15:55.076768 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.076743    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-default-certificate\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.076843 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.076770    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-stats-auth\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.083233 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.083206    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:55.084447 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.084429    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwj7\" (UniqueName: \"kubernetes.io/projected/e0f775f1-3efe-4432-b60e-f184557e7f60-kube-api-access-9zwj7\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.476857 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.476823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:55.477027 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.476962    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:55.477074 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.477031    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:56.477016776 +0000 UTC m=+62.146824122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:55.577837 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.577809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.577986 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:55.577871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:55.578066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.577999    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:55.578066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.578019    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:56.577999193 +0000 UTC m=+62.247806559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:55.578066 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:55.578051    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:56.57804 +0000 UTC m=+62.247847352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:15:56.483629 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:56.483588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:56.484101 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:56.483755    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:56.484101 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:56.483832    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:58.483812669 +0000 UTC m=+64.153620032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:56.584684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:56.584658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:56.584835 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:56.584741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:56.584835 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:56.584798    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:56.584916 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:56.584874    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:58.58485479 +0000 UTC m=+64.254662137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:15:56.584916 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:56.584889    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:58.584882962 +0000 UTC m=+64.254690308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:58.499344 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:58.499295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:15:58.499735 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:58.499453    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:58.499735 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:58.499516    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:02.499501458 +0000 UTC m=+68.169308804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:15:58.599919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:58.599889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:58.600107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:58.599977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:15:58.600107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:58.600058    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:02.600038534 +0000 UTC m=+68.269845880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:15:58.600107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:58.600069    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:15:58.600107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:15:58.600106    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:02.600095707 +0000 UTC m=+68.269903052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:15:59.796106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.796075    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"]
Jul 02 13:15:59.798598 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.798583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.801079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.801055    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 02 13:15:59.801079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.801065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f25bm\""
Jul 02 13:15:59.801079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.801065    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:15:59.802254 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.802236    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 02 13:15:59.802441 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.802424    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 02 13:15:59.806397 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.806376    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 02 13:15:59.808546 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.808524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-trusted-ca\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.808703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.808585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-config\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.808703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.808600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07174aea-ad00-466e-a362-cdd03e3c9c1c-serving-cert\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.808703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.808654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6v7\" (UniqueName: \"kubernetes.io/projected/07174aea-ad00-466e-a362-cdd03e3c9c1c-kube-api-access-kn6v7\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.809534 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.809515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"]
Jul 02 13:15:59.909834 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.909807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-config\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.910039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.909837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07174aea-ad00-466e-a362-cdd03e3c9c1c-serving-cert\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.910039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.909868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6v7\" (UniqueName: \"kubernetes.io/projected/07174aea-ad00-466e-a362-cdd03e3c9c1c-kube-api-access-kn6v7\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.910039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.909931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-trusted-ca\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.910383 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.910365    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-config\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.910697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.910680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07174aea-ad00-466e-a362-cdd03e3c9c1c-trusted-ca\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.912042 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.912027    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07174aea-ad00-466e-a362-cdd03e3c9c1c-serving-cert\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:15:59.922328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:15:59.922307    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6v7\" (UniqueName: \"kubernetes.io/projected/07174aea-ad00-466e-a362-cdd03e3c9c1c-kube-api-access-kn6v7\") pod \"console-operator-7fdbdf9c99-6pm9w\" (UID: \"07174aea-ad00-466e-a362-cdd03e3c9c1c\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:00.102578 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.102513    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w7plw"]
Jul 02 13:16:00.105590 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.105574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.107481 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.107463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:00.108070 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.108054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:16:00.108130 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.108068    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bdfl\""
Jul 02 13:16:00.108130 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.108061    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:16:00.108220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.108162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:16:00.108394 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.108382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:16:00.116763 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.116743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7plw"]
Jul 02 13:16:00.142077 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.142042    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:00.211763 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.211737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.211906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.211796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71251c71-6232-4757-8831-5794576e5a2a-crio-socket\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.211906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.211841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71251c71-6232-4757-8831-5794576e5a2a-data-volume\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.211906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.211896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.212053 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.211978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qll6m\" (UniqueName: \"kubernetes.io/projected/71251c71-6232-4757-8831-5794576e5a2a-kube-api-access-qll6m\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.223221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.223176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"]
Jul 02 13:16:00.226159 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:00.226136    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07174aea_ad00_466e_a362_cdd03e3c9c1c.slice/crio-9d4c3a7718efe7f3b657ac637ac32e5935201cd897c93890538aee915a7c163b WatchSource:0}: Error finding container 9d4c3a7718efe7f3b657ac637ac32e5935201cd897c93890538aee915a7c163b: Status 404 returned error can't find the container with id 9d4c3a7718efe7f3b657ac637ac32e5935201cd897c93890538aee915a7c163b
Jul 02 13:16:00.312459 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71251c71-6232-4757-8831-5794576e5a2a-crio-socket\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312541 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71251c71-6232-4757-8831-5794576e5a2a-data-volume\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312541 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qll6m\" (UniqueName: \"kubernetes.io/projected/71251c71-6232-4757-8831-5794576e5a2a-kube-api-access-qll6m\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312553    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71251c71-6232-4757-8831-5794576e5a2a-crio-socket\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312682 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.312682 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.312656    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:00.312742 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.312690    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:16:00.312742 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.312717    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.812696986 +0000 UTC m=+66.482504337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:00.312742 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.312736    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:00.812725354 +0000 UTC m=+66.482532699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : configmap "kube-rbac-proxy" not found
Jul 02 13:16:00.312840 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.312754    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71251c71-6232-4757-8831-5794576e5a2a-data-volume\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.323283 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.323264    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qll6m\" (UniqueName: \"kubernetes.io/projected/71251c71-6232-4757-8831-5794576e5a2a-kube-api-access-qll6m\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.513391 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.513326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:16:00.513391 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.513376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:16:00.513531 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.513427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:16:00.515809 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.515793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:16:00.515856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.515807    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:16:00.515856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.515807    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:16:00.523730 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.523711    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 13:16:00.523786 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.523766    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs podName:206cd077-8764-42f9-86ca-71d3cc81fa57 nodeName:}" failed. No retries permitted until 2026-07-02 13:17:04.523753475 +0000 UTC m=+130.193560826 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs") pod "network-metrics-daemon-ffcp9" (UID: "206cd077-8764-42f9-86ca-71d3cc81fa57") : secret "metrics-daemon-secret" not found
Jul 02 13:16:00.525712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.525694    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38b4b3cd-994a-47be-a765-884a6af449c8-original-pull-secret\") pod \"global-pull-secret-syncer-5d8ms\" (UID: \"38b4b3cd-994a-47be-a765-884a6af449c8\") " pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:16:00.525778 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.525723    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:16:00.535776 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.535756    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c65\" (UniqueName: \"kubernetes.io/projected/9f73406e-e073-47bb-b010-315f56ab60d8-kube-api-access-s4c65\") pod \"network-check-target-55ttw\" (UID: \"9f73406e-e073-47bb-b010-315f56ab60d8\") " pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:16:00.614597 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.614571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:00.614713 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.614696    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:16:00.614783 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.614773    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls podName:c2d57480-bc69-4de0-954a-098c7f6e0903 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:32.61475443 +0000 UTC m=+98.284561793 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls") pod "dns-default-4d98s" (UID: "c2d57480-bc69-4de0-954a-098c7f6e0903") : secret "dns-default-metrics-tls" not found
Jul 02 13:16:00.615982 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.615964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d8ms"
Jul 02 13:16:00.617079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.617059    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:00.624047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.624025    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ncrqc\""
Jul 02 13:16:00.632846 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.632830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:16:00.717697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.717657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:00.747491 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.747463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5d8ms"]
Jul 02 13:16:00.751612 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:00.751564    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b4b3cd_994a_47be_a765_884a6af449c8.slice/crio-cf8b3b50d9565ddeea152acd280d3a4df057d3a85a12c543e6ffa3655d7db3f3 WatchSource:0}: Error finding container cf8b3b50d9565ddeea152acd280d3a4df057d3a85a12c543e6ffa3655d7db3f3: Status 404 returned error can't find the container with id cf8b3b50d9565ddeea152acd280d3a4df057d3a85a12c543e6ffa3655d7db3f3
Jul 02 13:16:00.763038 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.763008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55ttw"]
Jul 02 13:16:00.765864 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:00.765809    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f73406e_e073_47bb_b010_315f56ab60d8.slice/crio-fa57d6b86a32be3c2dc61fb0cfb5562c14c3461897fbef7a7afe02180e5db408 WatchSource:0}: Error finding container fa57d6b86a32be3c2dc61fb0cfb5562c14c3461897fbef7a7afe02180e5db408: Status 404 returned error can't find the container with id fa57d6b86a32be3c2dc61fb0cfb5562c14c3461897fbef7a7afe02180e5db408
Jul 02 13:16:00.816590 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.816557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.816988 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.816689    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:00.816988 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.816734    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:16:00.816988 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.816748    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:01.816728804 +0000 UTC m=+67.486536167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:00.816988 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:00.816768    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:01.816757497 +0000 UTC m=+67.486564856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : configmap "kube-rbac-proxy" not found
Jul 02 13:16:00.816988 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.816691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:00.963101 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.963067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55ttw" event={"ID":"9f73406e-e073-47bb-b010-315f56ab60d8","Type":"ContainerStarted","Data":"fa57d6b86a32be3c2dc61fb0cfb5562c14c3461897fbef7a7afe02180e5db408"}
Jul 02 13:16:00.964121 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.964084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5d8ms" event={"ID":"38b4b3cd-994a-47be-a765-884a6af449c8","Type":"ContainerStarted","Data":"cf8b3b50d9565ddeea152acd280d3a4df057d3a85a12c543e6ffa3655d7db3f3"}
Jul 02 13:16:00.965238 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:00.965201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" event={"ID":"07174aea-ad00-466e-a362-cdd03e3c9c1c","Type":"ContainerStarted","Data":"9d4c3a7718efe7f3b657ac637ac32e5935201cd897c93890538aee915a7c163b"}
Jul 02 13:16:01.073268 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:01.073220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:01.193893 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:01.193853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:01.824151 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:01.824118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:01.824701 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:01.824180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:01.824701 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:01.824249    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:16:01.824701 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:01.824332    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:03.824310394 +0000 UTC m=+69.494117750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : configmap "kube-rbac-proxy" not found
Jul 02 13:16:01.824701 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:01.824343    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:01.824701 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:01.824392    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:03.824377987 +0000 UTC m=+69.494185344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:01.865302 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:01.865181    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v295b_91893a1e-ad7e-45b5-a53c-17c189ce5221/dns-node-resolver/0.log"
Jul 02 13:16:02.529529 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:02.529492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:02.529720 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:02.529646    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:02.529784 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:02.529727    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:10.529704825 +0000 UTC m=+76.199512177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:02.630285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:02.630252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:02.630442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:02.630346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:02.630442 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:02.630423    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:10.63039957 +0000 UTC m=+76.300206919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:16:02.630442 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:02.630440    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:16:02.630587 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:02.630479    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:10.630467595 +0000 UTC m=+76.300274975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:16:02.665154 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:02.665130    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rv7sq_8f0cc46a-0ad9-4952-bf16-b87d344e38ee/node-ca/0.log"
Jul 02 13:16:03.267911 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:03.267881    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vm7c9_b1311dc9-756f-41d6-8749-bfdbd03fe4cd/serve-healthcheck-canary/0.log"
Jul 02 13:16:03.841273 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:03.841238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:03.841428 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:03.841386    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:03.841428 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:03.841402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:03.841501 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:03.841443    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:07.84142801 +0000 UTC m=+73.511235357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:03.841545 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:03.841499    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:16:03.841582 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:03.841557    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:07.841541421 +0000 UTC m=+73.511348771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : configmap "kube-rbac-proxy" not found
Jul 02 13:16:04.457989 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:04.457948    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:04.558190 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:04.558154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:05.945919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.945876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:05.979741 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.979676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/0.log"
Jul 02 13:16:05.979741 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.979712    2565 generic.go:356] "Generic (PLEG): container finished" podID="07174aea-ad00-466e-a362-cdd03e3c9c1c" containerID="539deb2f50a885549ecf455cdd091b26cf025e9b9f85e3a4af7696822cf104e8" exitCode=255
Jul 02 13:16:05.979893 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.979814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" event={"ID":"07174aea-ad00-466e-a362-cdd03e3c9c1c","Type":"ContainerDied","Data":"539deb2f50a885549ecf455cdd091b26cf025e9b9f85e3a4af7696822cf104e8"}
Jul 02 13:16:05.980071 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.980053    2565 scope.go:117] "RemoveContainer" containerID="539deb2f50a885549ecf455cdd091b26cf025e9b9f85e3a4af7696822cf104e8"
Jul 02 13:16:05.981421 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.981390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55ttw" event={"ID":"9f73406e-e073-47bb-b010-315f56ab60d8","Type":"ContainerStarted","Data":"cec01600864772b7b83555d75a4848a8345f6d7e6e6ec64ce581ef4fe3ff2d2d"}
Jul 02 13:16:05.981527 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.981467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:16:05.982947 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:05.982917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5d8ms" event={"ID":"38b4b3cd-994a-47be-a765-884a6af449c8","Type":"ContainerStarted","Data":"7ac7de35a84d0bd82dd465d2fab71645c42601dfaba3e9ae3a69e1a9b6efc91f"}
Jul 02 13:16:06.004641 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.004614    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:06.012300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.012264    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5d8ms" podStartSLOduration=67.006404795 podStartE2EDuration="1m12.012248771s" totalImagesPullingTime="5.005843976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:06.011231715 +0000 UTC m=+71.681039084" watchObservedRunningTime="2026-07-02 13:16:06.012248771 +0000 UTC m=+71.682056174"
Jul 02 13:16:06.026974 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.026912    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-55ttw" podStartSLOduration=67.048895441 podStartE2EDuration="1m12.026899869s" totalImagesPullingTime="4.978004428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:06.026350844 +0000 UTC m=+71.696158216" watchObservedRunningTime="2026-07-02 13:16:06.026899869 +0000 UTC m=+71.696707236"
Jul 02 13:16:06.987085 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987059    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/1.log"
Jul 02 13:16:06.987442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/0.log"
Jul 02 13:16:06.987442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987438    2565 generic.go:356] "Generic (PLEG): container finished" podID="07174aea-ad00-466e-a362-cdd03e3c9c1c" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe" exitCode=255
Jul 02 13:16:06.987543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" event={"ID":"07174aea-ad00-466e-a362-cdd03e3c9c1c","Type":"ContainerDied","Data":"28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"}
Jul 02 13:16:06.987596 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987546    2565 scope.go:117] "RemoveContainer" containerID="539deb2f50a885549ecf455cdd091b26cf025e9b9f85e3a4af7696822cf104e8"
Jul 02 13:16:06.987731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:06.987712    2565 scope.go:117] "RemoveContainer" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"
Jul 02 13:16:06.987957 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:06.987911    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-6pm9w_openshift-console-operator(07174aea-ad00-466e-a362-cdd03e3c9c1c)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podUID="07174aea-ad00-466e-a362-cdd03e3c9c1c"
Jul 02 13:16:07.871483 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:07.871448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:07.871631 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:07.871497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:07.871631 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:07.871579    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:16:07.871631 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:07.871621    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:07.871728 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:07.871644    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:15.871629383 +0000 UTC m=+81.541436730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : configmap "kube-rbac-proxy" not found
Jul 02 13:16:07.871728 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:07.871660    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:15.871649802 +0000 UTC m=+81.541457153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:07.991213 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:07.991186    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/1.log"
Jul 02 13:16:07.991591 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:07.991494    2565 scope.go:117] "RemoveContainer" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"
Jul 02 13:16:07.991654 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:07.991634    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-6pm9w_openshift-console-operator(07174aea-ad00-466e-a362-cdd03e3c9c1c)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podUID="07174aea-ad00-466e-a362-cdd03e3c9c1c"
Jul 02 13:16:08.128831 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:08.128750    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:08.216742 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:08.216708    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:08.288204 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:08.288183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:10.108284 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.108253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:10.108665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.108309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:10.108665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.108588    2565 scope.go:117] "RemoveContainer" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"
Jul 02 13:16:10.108740 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.108722    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-6pm9w_openshift-console-operator(07174aea-ad00-466e-a362-cdd03e3c9c1c)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podUID="07174aea-ad00-466e-a362-cdd03e3c9c1c"
Jul 02 13:16:10.590831 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.590795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:10.591013 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.590921    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:10.591013 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.590991    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:26.59097727 +0000 UTC m=+92.260784616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:10.691890 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.691846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:10.692052 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:10.691923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:10.692052 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.692027    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:16:10.692135 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.692068    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:26.692055132 +0000 UTC m=+92.361862479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:16:10.692135 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:10.692084    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:26.692077027 +0000 UTC m=+92.361884372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : secret "router-metrics-certs-default" not found
Jul 02 13:16:11.284666 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.284636    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"]
Jul 02 13:16:11.287049 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.287034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.290676 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.290652    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 02 13:16:11.290791 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.290652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 13:16:11.290791 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.290701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 13:16:11.290791 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.290679    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 13:16:11.297174 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.297153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"]
Jul 02 13:16:11.395548 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.395517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986b33b9-bad6-4005-b800-e15015c75a33-tmp\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.395672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.395554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgssw\" (UniqueName: \"kubernetes.io/projected/986b33b9-bad6-4005-b800-e15015c75a33-kube-api-access-dgssw\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.395672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.395581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/986b33b9-bad6-4005-b800-e15015c75a33-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.496987 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.496948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986b33b9-bad6-4005-b800-e15015c75a33-tmp\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.496987 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.496992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgssw\" (UniqueName: \"kubernetes.io/projected/986b33b9-bad6-4005-b800-e15015c75a33-kube-api-access-dgssw\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.497157 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.497018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/986b33b9-bad6-4005-b800-e15015c75a33-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.497330 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.497308    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986b33b9-bad6-4005-b800-e15015c75a33-tmp\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.499332 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.499316    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/986b33b9-bad6-4005-b800-e15015c75a33-klusterlet-config\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.506055 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.506032    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgssw\" (UniqueName: \"kubernetes.io/projected/986b33b9-bad6-4005-b800-e15015c75a33-kube-api-access-dgssw\") pod \"klusterlet-addon-workmgr-f9c7bb796-ktsl7\" (UID: \"986b33b9-bad6-4005-b800-e15015c75a33\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.595373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.595304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:11.711535 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:11.711511    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"]
Jul 02 13:16:11.713904 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:11.713879    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986b33b9_bad6_4005_b800_e15015c75a33.slice/crio-0bc629d48a668a9e820d8bbd00180602ca47b3ef243aa902ce70e2b5262f4f5a WatchSource:0}: Error finding container 0bc629d48a668a9e820d8bbd00180602ca47b3ef243aa902ce70e2b5262f4f5a: Status 404 returned error can't find the container with id 0bc629d48a668a9e820d8bbd00180602ca47b3ef243aa902ce70e2b5262f4f5a
Jul 02 13:16:12.001115 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:12.001031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7" event={"ID":"986b33b9-bad6-4005-b800-e15015c75a33","Type":"ContainerStarted","Data":"0bc629d48a668a9e820d8bbd00180602ca47b3ef243aa902ce70e2b5262f4f5a"}
Jul 02 13:16:12.997166 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:12.997130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:13.961036 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:13.960987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:14.962684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:14.962614    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.063336 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.063302    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.398057 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.398028    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.498388 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.498351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.671092 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.671014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.771688 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.771649    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:15.934717 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.934649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:15.934717 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.934695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:15.934884 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:15.934872    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:15.934931 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:15.934924    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls podName:71251c71-6232-4757-8831-5794576e5a2a nodeName:}" failed. No retries permitted until 2026-07-02 13:16:31.934910612 +0000 UTC m=+97.604717959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7plw" (UID: "71251c71-6232-4757-8831-5794576e5a2a") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:16:15.935134 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:15.935119    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71251c71-6232-4757-8831-5794576e5a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:16.012971 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:16.012927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7" event={"ID":"986b33b9-bad6-4005-b800-e15015c75a33","Type":"ContainerStarted","Data":"ce170225b6f5ee8405c46095675e667fd9dd4643cebca9b087dba413a9934320"}
Jul 02 13:16:16.013293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:16.013181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:16.014750 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:16.014732    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7"
Jul 02 13:16:16.034160 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:16.034125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f9c7bb796-ktsl7" podStartSLOduration=1.621852777 podStartE2EDuration="5.034113076s" totalImagesPullingTime="3.412260299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:16.033337476 +0000 UTC m=+81.703144847" watchObservedRunningTime="2026-07-02 13:16:16.034113076 +0000 UTC m=+81.703920447"
Jul 02 13:16:17.273892 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.273859    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"]
Jul 02 13:16:17.278153 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.278132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.280723 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.280704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c6xb8\""
Jul 02 13:16:17.280815 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.280747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 13:16:17.281771 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.281744    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 13:16:17.281851 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.281780    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 13:16:17.281851 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.281785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 13:16:17.286391 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.286373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"]
Jul 02 13:16:17.315851 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.315824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:17.346843 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.346824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea44b4f2-cbce-4f20-8303-815334655611-signing-key\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.346920 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.346850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6hc\" (UniqueName: \"kubernetes.io/projected/ea44b4f2-cbce-4f20-8303-815334655611-kube-api-access-xx6hc\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.346920 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.346897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea44b4f2-cbce-4f20-8303-815334655611-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.372570 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.372541    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:17.447824 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.447807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea44b4f2-cbce-4f20-8303-815334655611-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.447913 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.447870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea44b4f2-cbce-4f20-8303-815334655611-signing-key\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.447913 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.447898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6hc\" (UniqueName: \"kubernetes.io/projected/ea44b4f2-cbce-4f20-8303-815334655611-kube-api-access-xx6hc\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.448408 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.448390    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea44b4f2-cbce-4f20-8303-815334655611-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.450202 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.450185    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea44b4f2-cbce-4f20-8303-815334655611-signing-key\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.457668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.457651    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6hc\" (UniqueName: \"kubernetes.io/projected/ea44b4f2-cbce-4f20-8303-815334655611-kube-api-access-xx6hc\") pod \"service-ca-7bb4f4bcc7-6bhdf\" (UID: \"ea44b4f2-cbce-4f20-8303-815334655611\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.587085 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.587027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"
Jul 02 13:16:17.702713 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:17.702684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf"]
Jul 02 13:16:17.705698 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:17.705672    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea44b4f2_cbce_4f20_8303_815334655611.slice/crio-3712dbac5d60464ff0470f1e0cb89f00ee89ae01a2a577d6693f08d7c21bb2b9 WatchSource:0}: Error finding container 3712dbac5d60464ff0470f1e0cb89f00ee89ae01a2a577d6693f08d7c21bb2b9: Status 404 returned error can't find the container with id 3712dbac5d60464ff0470f1e0cb89f00ee89ae01a2a577d6693f08d7c21bb2b9
Jul 02 13:16:18.019300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:18.019216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf" event={"ID":"ea44b4f2-cbce-4f20-8303-815334655611","Type":"ContainerStarted","Data":"3712dbac5d60464ff0470f1e0cb89f00ee89ae01a2a577d6693f08d7c21bb2b9"}
Jul 02 13:16:18.981189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:18.981153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:20.030589 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:20.030546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf" event={"ID":"ea44b4f2-cbce-4f20-8303-815334655611","Type":"ContainerStarted","Data":"d9d443ee8f5bb7728469ad50d470c78042bb20cf9b2e3129bda0a9bdf3f38258"}
Jul 02 13:16:20.047401 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:20.047362    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bhdf" podStartSLOduration=1.029773884 podStartE2EDuration="3.047348227s" totalImagesPullingTime="2.017574343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:20.046954219 +0000 UTC m=+85.716761588" watchObservedRunningTime="2026-07-02 13:16:20.047348227 +0000 UTC m=+85.717155597"
Jul 02 13:16:20.701838 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:20.701805    2565 scope.go:117] "RemoveContainer" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"
Jul 02 13:16:21.035417 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.035388    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:16:21.035785 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.035770    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/1.log"
Jul 02 13:16:21.035827 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.035805    2565 generic.go:356] "Generic (PLEG): container finished" podID="07174aea-ad00-466e-a362-cdd03e3c9c1c" containerID="663b37a0da5414c8d2ded64a444ecafb156cc0442aaaf02b876a6c493d891012" exitCode=255
Jul 02 13:16:21.035907 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.035886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" event={"ID":"07174aea-ad00-466e-a362-cdd03e3c9c1c","Type":"ContainerDied","Data":"663b37a0da5414c8d2ded64a444ecafb156cc0442aaaf02b876a6c493d891012"}
Jul 02 13:16:21.035977 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.035964    2565 scope.go:117] "RemoveContainer" containerID="28877b9f5320d82774a237311a8146af43fc8360b43e92df03a8fc85fac7fffe"
Jul 02 13:16:21.036326 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.036296    2565 scope.go:117] "RemoveContainer" containerID="663b37a0da5414c8d2ded64a444ecafb156cc0442aaaf02b876a6c493d891012"
Jul 02 13:16:21.036474 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:21.036455    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-6pm9w_openshift-console-operator(07174aea-ad00-466e-a362-cdd03e3c9c1c)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podUID="07174aea-ad00-466e-a362-cdd03e3c9c1c"
Jul 02 13:16:21.609360 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.609323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:21.703543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.703505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:21.985730 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:21.985637    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.041328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:22.041294    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:16:22.136533 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:22.136448    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.265630 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:22.265523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.897951 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:22.897909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.995209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:22.995178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:23.099062 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:23.099029    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:23.386697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:23.386669    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:24.040838 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:24.040804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:24.101085 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:24.101055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:24.182199 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:24.182171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:24.582344 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:24.582312    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:24.779676 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:24.779633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:25.009123 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:25.009034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:25.365469 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:25.365439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:25.466289 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:25.466265    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:25.587572 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:25.587540    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:25.979819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:25.979786    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:26.036106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.036072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:26.580810 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.580775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:26.620733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.620705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:26.620833 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:26.620816    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:26.620923 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:26.620871    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls podName:a639313a-e342-4393-a017-c9ee4dd9da02 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:58.620858531 +0000 UTC m=+124.290665876 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-vm8xz" (UID: "a639313a-e342-4393-a017-c9ee4dd9da02") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:16:26.721649 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.721623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:26.721760 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.721671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:26.721829 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:26.721812    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle podName:e0f775f1-3efe-4432-b60e-f184557e7f60 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:58.72178877 +0000 UTC m=+124.391596129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle") pod "router-default-58f7cb747c-qh9k6" (UID: "e0f775f1-3efe-4432-b60e-f184557e7f60") : configmap references non-existent config key: service-ca.crt
Jul 02 13:16:26.723995 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:26.723970    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f775f1-3efe-4432-b60e-f184557e7f60-metrics-certs\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:27.059859 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.059821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.118170 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.118142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.260186 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.260157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.384005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.383913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.766239 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.766150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.991140 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:27.991108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:28.393921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:28.393886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:28.586901 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:28.586868    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:29.181972 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:29.181921    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:29.280190 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:29.280152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:29.585921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:29.585889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:30.108210 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.108176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:30.108370 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.108273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:30.108658 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.108645    2565 scope.go:117] "RemoveContainer" containerID="663b37a0da5414c8d2ded64a444ecafb156cc0442aaaf02b876a6c493d891012"
Jul 02 13:16:30.108843 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:16:30.108820    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-6pm9w_openshift-console-operator(07174aea-ad00-466e-a362-cdd03e3c9c1c)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podUID="07174aea-ad00-466e-a362-cdd03e3c9c1c"
Jul 02 13:16:30.122597 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.122565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:30.490121 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.490040    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:30.787985 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:30.787954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:31.182470 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:31.182385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:31.580446 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:31.580407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:31.961277 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:31.961194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:31.963347 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:31.963330    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71251c71-6232-4757-8831-5794576e5a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7plw\" (UID: \"71251c71-6232-4757-8831-5794576e5a2a\") " pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:31.984578 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:31.984550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:32.214992 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.214903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7plw"
Jul 02 13:16:32.333357 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.333333    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7plw"]
Jul 02 13:16:32.336113 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:32.336086    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71251c71_6232_4757_8831_5794576e5a2a.slice/crio-c614c3804878433e782809fd8a78f057c157e68f3cb982c202f66329da076b87 WatchSource:0}: Error finding container c614c3804878433e782809fd8a78f057c157e68f3cb982c202f66329da076b87: Status 404 returned error can't find the container with id c614c3804878433e782809fd8a78f057c157e68f3cb982c202f66329da076b87
Jul 02 13:16:32.561253 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.561227    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:32.628462 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.628433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:32.666761 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.666732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:32.669070 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.669052    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d57480-bc69-4de0-954a-098c7f6e0903-metrics-tls\") pod \"dns-default-4d98s\" (UID: \"c2d57480-bc69-4de0-954a-098c7f6e0903\") " pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:32.963021 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.962991    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrpc5\""
Jul 02 13:16:32.971137 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:32.971115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:33.076786 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:33.076754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7plw" event={"ID":"71251c71-6232-4757-8831-5794576e5a2a","Type":"ContainerStarted","Data":"c614c3804878433e782809fd8a78f057c157e68f3cb982c202f66329da076b87"}
Jul 02 13:16:33.089377 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:33.089355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4d98s"]
Jul 02 13:16:33.091353 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:33.091331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d57480_bc69_4de0_954a_098c7f6e0903.slice/crio-954073f8db3054120299a74d5037a87e835152882401aeb31528c608d1cb1e57 WatchSource:0}: Error finding container 954073f8db3054120299a74d5037a87e835152882401aeb31528c608d1cb1e57: Status 404 returned error can't find the container with id 954073f8db3054120299a74d5037a87e835152882401aeb31528c608d1cb1e57
Jul 02 13:16:34.083474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:34.083383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d98s" event={"ID":"c2d57480-bc69-4de0-954a-098c7f6e0903","Type":"ContainerStarted","Data":"954073f8db3054120299a74d5037a87e835152882401aeb31528c608d1cb1e57"}
Jul 02 13:16:36.989586 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:36.989559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-55ttw"
Jul 02 13:16:38.097518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.097487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d98s" event={"ID":"c2d57480-bc69-4de0-954a-098c7f6e0903","Type":"ContainerStarted","Data":"ce276996525e2b2de204a8baceabb454e5e0c3e330d4a901fa9e73e503aa5ef1"}
Jul 02 13:16:38.097914 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.097548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d98s" event={"ID":"c2d57480-bc69-4de0-954a-098c7f6e0903","Type":"ContainerStarted","Data":"17170fdd36956fe1b485f00b300fab4902df64cda548903bd9f3345067e320d7"}
Jul 02 13:16:38.097914 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.097578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:38.099004 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.098981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7plw" event={"ID":"71251c71-6232-4757-8831-5794576e5a2a","Type":"ContainerStarted","Data":"f2d5f53eee887c326dc78d4146d2588e86b1938eef80adf4a13c7debb66f3634"}
Jul 02 13:16:38.115324 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.115284    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4d98s" podStartSLOduration=65.578175989 podStartE2EDuration="1m10.115270959s" totalImagesPullingTime="4.53709497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:38.114093789 +0000 UTC m=+103.783901158" watchObservedRunningTime="2026-07-02 13:16:38.115270959 +0000 UTC m=+103.785078328"
Jul 02 13:16:38.741191 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.741166    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85d4f8f779-ncqr4"]
Jul 02 13:16:38.744049 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.744034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.748092 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.748073    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 13:16:38.748185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.748156    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 13:16:38.748877 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.748854    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 13:16:38.749045 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.749026    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pfv2v\""
Jul 02 13:16:38.754321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.754306    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 13:16:38.758336 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.758317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d4f8f779-ncqr4"]
Jul 02 13:16:38.916668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-trusted-ca\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.916668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qbg\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-kube-api-access-64qbg\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.916841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-certificates\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.916841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-installation-pull-secrets\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.916841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-tls\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.917013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-bound-sa-token\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.917013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-image-registry-private-configuration\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:38.917013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:38.916892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6151020-9cdd-4b7a-a544-60dcc396d622-ca-trust-extracted\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017383 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6151020-9cdd-4b7a-a544-60dcc396d622-ca-trust-extracted\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017383 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-trusted-ca\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qbg\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-kube-api-access-64qbg\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-certificates\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-installation-pull-secrets\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-tls\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-bound-sa-token\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017867 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-image-registry-private-configuration\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.017867 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.017747    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6151020-9cdd-4b7a-a544-60dcc396d622-ca-trust-extracted\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.018387 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.018365    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-trusted-ca\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.018519 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.018373    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-certificates\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.020003 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.019981    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-registry-tls\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.020125 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.020079    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-image-registry-private-configuration\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.020360 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.020345    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6151020-9cdd-4b7a-a544-60dcc396d622-installation-pull-secrets\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.028193 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.028172    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qbg\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-kube-api-access-64qbg\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.032529 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.032511    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6151020-9cdd-4b7a-a544-60dcc396d622-bound-sa-token\") pod \"image-registry-85d4f8f779-ncqr4\" (UID: \"b6151020-9cdd-4b7a-a544-60dcc396d622\") " pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.053602 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.053579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:39.105719 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.105636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7plw" event={"ID":"71251c71-6232-4757-8831-5794576e5a2a","Type":"ContainerStarted","Data":"afaa1f04a51aa367ce370568fb8676a012c9168fd012d84614da725c70921e1b"}
Jul 02 13:16:39.189492 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.189408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d4f8f779-ncqr4"]
Jul 02 13:16:39.192537 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:39.192470    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6151020_9cdd_4b7a_a544_60dcc396d622.slice/crio-452e6bd7a435a964ebc85d96c034d8456cd9199704628c952b3e5b6714edcdc9 WatchSource:0}: Error finding container 452e6bd7a435a964ebc85d96c034d8456cd9199704628c952b3e5b6714edcdc9: Status 404 returned error can't find the container with id 452e6bd7a435a964ebc85d96c034d8456cd9199704628c952b3e5b6714edcdc9
Jul 02 13:16:39.241853 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.241821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:39.343011 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:39.342975    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:40.111825 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:40.111790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4" event={"ID":"b6151020-9cdd-4b7a-a544-60dcc396d622","Type":"ContainerStarted","Data":"a0b31305031f2c50b59373f7a4820e6c0bfe398cc84cf4390a9808add0c137f5"}
Jul 02 13:16:40.112294 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:40.111841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4" event={"ID":"b6151020-9cdd-4b7a-a544-60dcc396d622","Type":"ContainerStarted","Data":"452e6bd7a435a964ebc85d96c034d8456cd9199704628c952b3e5b6714edcdc9"}
Jul 02 13:16:40.112294 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:40.111890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:16:40.140413 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:40.138237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4" podStartSLOduration=2.138217177 podStartE2EDuration="2.138217177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:40.136457825 +0000 UTC m=+105.806265193" watchObservedRunningTime="2026-07-02 13:16:40.138217177 +0000 UTC m=+105.808024547"
Jul 02 13:16:41.812140 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:41.812101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:41.912779 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:41.912746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:42.120395 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:42.120308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7plw" event={"ID":"71251c71-6232-4757-8831-5794576e5a2a","Type":"ContainerStarted","Data":"f557f3c06607d92a0067ca24754458b54db0515f621b62d078a1a1ff56b97f18"}
Jul 02 13:16:42.142924 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:42.142878    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w7plw" podStartSLOduration=33.439779425 podStartE2EDuration="42.142863841s" totalImagesPullingTime="8.703084416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:16:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:42.14206497 +0000 UTC m=+107.811872340" watchObservedRunningTime="2026-07-02 13:16:42.142863841 +0000 UTC m=+107.812671261"
Jul 02 13:16:44.703764 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:44.703729    2565 scope.go:117] "RemoveContainer" containerID="663b37a0da5414c8d2ded64a444ecafb156cc0442aaaf02b876a6c493d891012"
Jul 02 13:16:45.131836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.131807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:16:45.132004 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.131881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" event={"ID":"07174aea-ad00-466e-a362-cdd03e3c9c1c","Type":"ContainerStarted","Data":"474a16fa39be2f393d1bc50fc96d243eaaeeecec46fbd92c6130e5c2b21f337d"}
Jul 02 13:16:45.132194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.132169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:45.150364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.150322    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w" podStartSLOduration=40.628440161 podStartE2EDuration="46.150308194s" totalImagesPullingTime="5.521868033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:45.148552811 +0000 UTC m=+110.818360179" watchObservedRunningTime="2026-07-02 13:16:45.150308194 +0000 UTC m=+110.820115566"
Jul 02 13:16:45.341091 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.341054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-6pm9w"
Jul 02 13:16:45.542629 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.542595    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:45.614086 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.614045    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:45.719223 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.719177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:45.834681 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:45.834603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:46.007890 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:46.007860    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:46.203858 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:46.203785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:46.598361 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:46.598312    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:46.700071 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:46.700040    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:46.997842 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:46.997765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.098876 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.098847    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.425448 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.425417    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.616321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.616293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.740165 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.740081    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.840322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.840286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:47.899920 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:47.899894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:48.107485 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:48.107463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4d98s"
Jul 02 13:16:48.230500 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:48.230461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:48.327524 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:48.327476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:48.785817 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:48.785787    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:48.848903 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:48.848850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:54.637211 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:54.637175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:54.696476 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:54.696444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:56.250452 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:56.250415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:56.317826 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:56.317798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:56.778719 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:56.778686    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:56.835281 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:56.835246    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:57.769495 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:57.769460    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:57.828731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:57.828693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:58.034949 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.034912    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:58.136559 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.136525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:58.657172 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.657140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:58.659539 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.659508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a639313a-e342-4393-a017-c9ee4dd9da02-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-vm8xz\" (UID: \"a639313a-e342-4393-a017-c9ee4dd9da02\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:58.691868 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.691838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ff6bd\""
Jul 02 13:16:58.699816 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.699802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"
Jul 02 13:16:58.757543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.757516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:58.758337 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.758313    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0f775f1-3efe-4432-b60e-f184557e7f60-service-ca-bundle\") pod \"router-default-58f7cb747c-qh9k6\" (UID: \"e0f775f1-3efe-4432-b60e-f184557e7f60\") " pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:58.784502 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.784477    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wpnt9\""
Jul 02 13:16:58.792589 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.792567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:58.829806 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.829782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz"]
Jul 02 13:16:58.832874 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:58.832847    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda639313a_e342_4393_a017_c9ee4dd9da02.slice/crio-c44de2a8fc967e01473dd3a58fb69c5f925dbc86e4d17d7acc503ed2bc544c47 WatchSource:0}: Error finding container c44de2a8fc967e01473dd3a58fb69c5f925dbc86e4d17d7acc503ed2bc544c47: Status 404 returned error can't find the container with id c44de2a8fc967e01473dd3a58fb69c5f925dbc86e4d17d7acc503ed2bc544c47
Jul 02 13:16:58.903198 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.903168    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:58.920287 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:58.920132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58f7cb747c-qh9k6"]
Jul 02 13:16:58.922481 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:16:58.922456    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f775f1_3efe_4432_b60e_f184557e7f60.slice/crio-184648e8b2f7a32ee57839e57b00d4719752faf124cf6ffe9d6695be7af8eea8 WatchSource:0}: Error finding container 184648e8b2f7a32ee57839e57b00d4719752faf124cf6ffe9d6695be7af8eea8: Status 404 returned error can't find the container with id 184648e8b2f7a32ee57839e57b00d4719752faf124cf6ffe9d6695be7af8eea8
Jul 02 13:16:59.005874 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.005845    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:59.173363 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.173268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58f7cb747c-qh9k6" event={"ID":"e0f775f1-3efe-4432-b60e-f184557e7f60","Type":"ContainerStarted","Data":"704dd3b8ba7e61e8313e8c475dc07dd0de39d9aeab9b013cba22e5fe90f88959"}
Jul 02 13:16:59.173363 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.173328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58f7cb747c-qh9k6" event={"ID":"e0f775f1-3efe-4432-b60e-f184557e7f60","Type":"ContainerStarted","Data":"184648e8b2f7a32ee57839e57b00d4719752faf124cf6ffe9d6695be7af8eea8"}
Jul 02 13:16:59.174323 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.174300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz" event={"ID":"a639313a-e342-4393-a017-c9ee4dd9da02","Type":"ContainerStarted","Data":"c44de2a8fc967e01473dd3a58fb69c5f925dbc86e4d17d7acc503ed2bc544c47"}
Jul 02 13:16:59.191294 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.191249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-58f7cb747c-qh9k6" podStartSLOduration=65.191232984 podStartE2EDuration="1m5.191232984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:59.190690272 +0000 UTC m=+124.860497641" watchObservedRunningTime="2026-07-02 13:16:59.191232984 +0000 UTC m=+124.861040329"
Jul 02 13:16:59.793767 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.793734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:16:59.796210 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:16:59.796191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:17:00.178246 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:00.178164    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:17:00.179422 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:00.179401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58f7cb747c-qh9k6"
Jul 02 13:17:01.121056 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.121027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85d4f8f779-ncqr4"
Jul 02 13:17:01.284295 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.284259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:01.408819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.408736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:01.510541 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.510499    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:01.649949 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.649915    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:01.750660 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:01.750578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:02.180464 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:02.180434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:02.186279 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:02.186251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz" event={"ID":"a639313a-e342-4393-a017-c9ee4dd9da02","Type":"ContainerStarted","Data":"540d043b449c57e51e3d48bde6fd15b84ca3f6652476320a4f43c3073a1251ef"}
Jul 02 13:17:02.204144 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:02.204106    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-vm8xz" podStartSLOduration=65.780800294 podStartE2EDuration="1m8.204092592s" totalImagesPullingTime="2.423292298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:02.203057416 +0000 UTC m=+127.872864783" watchObservedRunningTime="2026-07-02 13:17:02.204092592 +0000 UTC m=+127.873899960"
Jul 02 13:17:02.362396 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:02.362346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:04.603184 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.603152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:17:04.605323 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.605301    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206cd077-8764-42f9-86ca-71d3cc81fa57-metrics-certs\") pod \"network-metrics-daemon-ffcp9\" (UID: \"206cd077-8764-42f9-86ca-71d3cc81fa57\") " pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:17:04.717739 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.717710    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qblmr\""
Jul 02 13:17:04.723122 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.723104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ffcp9"
Jul 02 13:17:04.841623 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.841591    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:04.846371 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.846350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ffcp9"]
Jul 02 13:17:04.848636 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:17:04.848611    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206cd077_8764_42f9_86ca_71d3cc81fa57.slice/crio-49824137a0db0ebf1f4eb9db9e47c75c4d7085f72954c9a5e809ace312ecc605 WatchSource:0}: Error finding container 49824137a0db0ebf1f4eb9db9e47c75c4d7085f72954c9a5e809ace312ecc605: Status 404 returned error can't find the container with id 49824137a0db0ebf1f4eb9db9e47c75c4d7085f72954c9a5e809ace312ecc605
Jul 02 13:17:04.945333 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:04.945292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:05.196834 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:05.196760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ffcp9" event={"ID":"206cd077-8764-42f9-86ca-71d3cc81fa57","Type":"ContainerStarted","Data":"49824137a0db0ebf1f4eb9db9e47c75c4d7085f72954c9a5e809ace312ecc605"}
Jul 02 13:17:05.851052 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:05.851015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:05.952919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:05.952884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.074317 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:06.074282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.176614 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:06.176492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.798424 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:06.798391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.897232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:06.897204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:07.205581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.205487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ffcp9" event={"ID":"206cd077-8764-42f9-86ca-71d3cc81fa57","Type":"ContainerStarted","Data":"2ae84633898fcf4dd644fb4d00d456b628d8b2928f15bf7db2576bbf179064ab"}
Jul 02 13:17:07.205581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.205546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ffcp9" event={"ID":"206cd077-8764-42f9-86ca-71d3cc81fa57","Type":"ContainerStarted","Data":"7edbbc9bfa3ed4a83444a905f0bdcd6a65c7365f8cc5484c3ee0865473291530"}
Jul 02 13:17:07.224947 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.224891    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ffcp9" podStartSLOduration=131.802648377 podStartE2EDuration="2m13.224874428s" totalImagesPullingTime="1.422226051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:07.223709234 +0000 UTC m=+132.893516602" watchObservedRunningTime="2026-07-02 13:17:07.224874428 +0000 UTC m=+132.894681795"
Jul 02 13:17:07.243306 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.243274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:07.838695 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.838660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:07.898321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:07.898286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:09.044794 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:09.044752    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:09.151094 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:09.151061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:09.649114 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:09.649072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:09.747983 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:09.747946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:10.228909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.228876    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:17:10.246769 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.246736    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:17:10.247171 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.246838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.249183 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.249159    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-s7qpc\""
Jul 02 13:17:10.249313 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.249220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:17:10.249313 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.249260    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:17:10.250353 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.250335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:17:10.250490 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.250472    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:17:10.250717 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.250694    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:17:10.250839 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.250715    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:17:10.250839 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.250789    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:17:10.255337 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.255316    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:17:10.347383 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347486 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347486 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347486 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7gx\" (UniqueName: \"kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347656 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347656 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.347656 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.347575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7gx\" (UniqueName: \"kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.448811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.448800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.449415 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.449392    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.449524 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.449504    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.449569 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.449508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.449625 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.449606    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.451104 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.451086    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.451215 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.451199    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.459107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.459084    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7gx\" (UniqueName: \"kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx\") pod \"console-5bc8df7d7b-sv9p8\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") " pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.556832 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.556807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:10.682888 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:10.682863    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:17:10.685188 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:17:10.685162    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db4f73_b762_4472_a2ee_359225bb9f99.slice/crio-c6e44ce60cef86ba3775f51ad941cafc3f47e117ea3a9f499bf9936fa5294e6f WatchSource:0}: Error finding container c6e44ce60cef86ba3775f51ad941cafc3f47e117ea3a9f499bf9936fa5294e6f: Status 404 returned error can't find the container with id c6e44ce60cef86ba3775f51ad941cafc3f47e117ea3a9f499bf9936fa5294e6f
Jul 02 13:17:11.220724 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:11.220683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8df7d7b-sv9p8" event={"ID":"f3db4f73-b762-4472-a2ee-359225bb9f99","Type":"ContainerStarted","Data":"c6e44ce60cef86ba3775f51ad941cafc3f47e117ea3a9f499bf9936fa5294e6f"}
Jul 02 13:17:11.265186 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:11.265151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:11.817522 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:11.817473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:11.882316 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:11.882274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:11.981891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:11.981857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:13.055575 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:13.055535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:13.119035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:13.119003    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:14.236463 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:14.233087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8df7d7b-sv9p8" event={"ID":"f3db4f73-b762-4472-a2ee-359225bb9f99","Type":"ContainerStarted","Data":"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"}
Jul 02 13:17:14.252181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:14.252109    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc8df7d7b-sv9p8" podStartSLOduration=1.620336612 podStartE2EDuration="4.252090663s" totalImagesPullingTime="2.631754051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:14.251008642 +0000 UTC m=+139.920816011" watchObservedRunningTime="2026-07-02 13:17:14.252090663 +0000 UTC m=+139.921898031"
Jul 02 13:17:15.438310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:15.438273    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.000620 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.000587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.100052 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.100015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.157309 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.157278    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.296390 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.296358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.401615 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.401554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:16.465790 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:16.465757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:17.978896 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:17.978861    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.195913 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.195876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.227669 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.227636    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"]
Jul 02 13:17:18.234521 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.234468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.237179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.237157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 13:17:18.237285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.237159    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:17:18.237285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.237209    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ktllc\""
Jul 02 13:17:18.238573 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.238555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:17:18.241593 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.241570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"]
Jul 02 13:17:18.251409 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.251387    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-w7lzm"]
Jul 02 13:17:18.261864 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.261846    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xdbs7"]
Jul 02 13:17:18.263975 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.263918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.266417 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.265972    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-w7lzm"]
Jul 02 13:17:18.266417 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.266035    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.267438 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.267421    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:17:18.268610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.268283    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 13:17:18.269819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.268325    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z2b4w\""
Jul 02 13:17:18.269997 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.268374    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 13:17:18.270085 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.268452    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:17:18.270161 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.269560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rpwj4\""
Jul 02 13:17:18.270256 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.269593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:17:18.270323 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.269614    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:17:18.295972 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.295948    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.308408 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.308518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj48g\" (UniqueName: \"kubernetes.io/projected/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-api-access-kj48g\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.308518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-sys\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.308518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-root\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.308518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88fec06-a219-4b2e-8e56-f71a07250ee1-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-wtmp\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-textfile\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-tls\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.308709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gxv\" (UniqueName: \"kubernetes.io/projected/e88fec06-a219-4b2e-8e56-f71a07250ee1-kube-api-access-64gxv\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.309078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.309078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-metrics-client-ca\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.309078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.309078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.308987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.309078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.309047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbg8\" (UniqueName: \"kubernetes.io/projected/eba184bd-079b-4132-9470-69224d9eed57-kube-api-access-pqbg8\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.309321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.309087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.309321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.309136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-metrics-client-ca\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.355503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.355482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.409909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.409887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-wtmp\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.409918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.410047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.409962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.410047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-textfile\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-tls\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gxv\" (UniqueName: \"kubernetes.io/projected/e88fec06-a219-4b2e-8e56-f71a07250ee1-kube-api-access-64gxv\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410074    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-wtmp\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-metrics-client-ca\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:17:18.410154    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.410247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbg8\" (UniqueName: \"kubernetes.io/projected/eba184bd-079b-4132-9470-69224d9eed57-kube-api-access-pqbg8\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-metrics-client-ca\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj48g\" (UniqueName: \"kubernetes.io/projected/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-api-access-kj48g\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410371    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-textfile\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:17:18.410429    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls podName:d1e83c05-f5fc-4f05-92f3-0e41c86fc624 nodeName:}" failed. No retries permitted until 2026-07-02 13:17:18.910391048 +0000 UTC m=+144.580198403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-w7lzm" (UID: "d1e83c05-f5fc-4f05-92f3-0e41c86fc624") : secret "kube-state-metrics-tls" not found
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-sys\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-root\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88fec06-a219-4b2e-8e56-f71a07250ee1-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.410619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.411148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410806    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.411148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410927    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.411148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.410991    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-metrics-client-ca\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.411148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.411023    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-root\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.411148 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:17:18.411107    2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 02 13:17:18.411386 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:17:18.411164    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls podName:e88fec06-a219-4b2e-8e56-f71a07250ee1 nodeName:}" failed. No retries permitted until 2026-07-02 13:17:18.911149362 +0000 UTC m=+144.580956708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls") pod "openshift-state-metrics-57f99c8b49-2lrbn" (UID: "e88fec06-a219-4b2e-8e56-f71a07250ee1") : secret "openshift-state-metrics-tls" not found
Jul 02 13:17:18.411386 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.411337    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.411663 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.411640    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88fec06-a219-4b2e-8e56-f71a07250ee1-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.411797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.411777    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-metrics-client-ca\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.411904 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.411849    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba184bd-079b-4132-9470-69224d9eed57-sys\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.413335 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.413311    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.413695 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.413675    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.413803 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.413784    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.414259 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.414239    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eba184bd-079b-4132-9470-69224d9eed57-node-exporter-tls\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.419040 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.419019    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbg8\" (UniqueName: \"kubernetes.io/projected/eba184bd-079b-4132-9470-69224d9eed57-kube-api-access-pqbg8\") pod \"node-exporter-xdbs7\" (UID: \"eba184bd-079b-4132-9470-69224d9eed57\") " pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.419116 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.419097    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj48g\" (UniqueName: \"kubernetes.io/projected/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-api-access-kj48g\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.420356 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.420336    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gxv\" (UniqueName: \"kubernetes.io/projected/e88fec06-a219-4b2e-8e56-f71a07250ee1-kube-api-access-64gxv\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.460683 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.460644    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.569277 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.569235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.586597 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.586568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xdbs7"
Jul 02 13:17:18.598403 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:17:18.598382    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba184bd_079b_4132_9470_69224d9eed57.slice/crio-2372a75617a032f46357a691638c024d1d2a2f5d4560badf9484a5aecb310a7b WatchSource:0}: Error finding container 2372a75617a032f46357a691638c024d1d2a2f5d4560badf9484a5aecb310a7b: Status 404 returned error can't find the container with id 2372a75617a032f46357a691638c024d1d2a2f5d4560badf9484a5aecb310a7b
Jul 02 13:17:18.655915 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.655884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.915804 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.915706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:18.915804 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.915765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.918326 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.918300    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1e83c05-f5fc-4f05-92f3-0e41c86fc624-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-w7lzm\" (UID: \"d1e83c05-f5fc-4f05-92f3-0e41c86fc624\") " pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:18.918416 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:18.918300    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e88fec06-a219-4b2e-8e56-f71a07250ee1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-2lrbn\" (UID: \"e88fec06-a219-4b2e-8e56-f71a07250ee1\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:19.157482 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.157442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"
Jul 02 13:17:19.177681 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.177658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm"
Jul 02 13:17:19.263976 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.263923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdbs7" event={"ID":"eba184bd-079b-4132-9470-69224d9eed57","Type":"ContainerStarted","Data":"2372a75617a032f46357a691638c024d1d2a2f5d4560badf9484a5aecb310a7b"}
Jul 02 13:17:19.282924 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.282863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:19.312885 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.312859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn"]
Jul 02 13:17:19.315418 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:17:19.315382    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88fec06_a219_4b2e_8e56_f71a07250ee1.slice/crio-17c393b019da250ba4db8a3354cd87c9940d9e602b966d7697a8275d10b45eac WatchSource:0}: Error finding container 17c393b019da250ba4db8a3354cd87c9940d9e602b966d7697a8275d10b45eac: Status 404 returned error can't find the container with id 17c393b019da250ba4db8a3354cd87c9940d9e602b966d7697a8275d10b45eac
Jul 02 13:17:19.332116 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.332089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-w7lzm"]
Jul 02 13:17:19.333915 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:17:19.333879    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e83c05_f5fc_4f05_92f3_0e41c86fc624.slice/crio-828553ea20f0be1ad1a0d837ca638015eedc98f865ee438ee122608e37b8c472 WatchSource:0}: Error finding container 828553ea20f0be1ad1a0d837ca638015eedc98f865ee438ee122608e37b8c472: Status 404 returned error can't find the container with id 828553ea20f0be1ad1a0d837ca638015eedc98f865ee438ee122608e37b8c472
Jul 02 13:17:19.360144 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.360118    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:19.947428 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:19.947379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:20.270247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.270163    2565 generic.go:356] "Generic (PLEG): container finished" podID="eba184bd-079b-4132-9470-69224d9eed57" containerID="5c0651ffc367b337208fdfb41e3c5d8245d7e434041e72aa0ab545908552a435" exitCode=0
Jul 02 13:17:20.270247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.270224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdbs7" event={"ID":"eba184bd-079b-4132-9470-69224d9eed57","Type":"ContainerDied","Data":"5c0651ffc367b337208fdfb41e3c5d8245d7e434041e72aa0ab545908552a435"}
Jul 02 13:17:20.271544 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.271517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm" event={"ID":"d1e83c05-f5fc-4f05-92f3-0e41c86fc624","Type":"ContainerStarted","Data":"828553ea20f0be1ad1a0d837ca638015eedc98f865ee438ee122608e37b8c472"}
Jul 02 13:17:20.273814 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.273789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn" event={"ID":"e88fec06-a219-4b2e-8e56-f71a07250ee1","Type":"ContainerStarted","Data":"b6900c402ed95be2a6cc30afba6263dc53cefef7d2902b4d12e46e37d6adb3f5"}
Jul 02 13:17:20.273912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.273846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn" event={"ID":"e88fec06-a219-4b2e-8e56-f71a07250ee1","Type":"ContainerStarted","Data":"06be0094822965f174e559badae20461eae9406c489b479ad9b27d6c640573d2"}
Jul 02 13:17:20.273912 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.273880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn" event={"ID":"e88fec06-a219-4b2e-8e56-f71a07250ee1","Type":"ContainerStarted","Data":"17c393b019da250ba4db8a3354cd87c9940d9e602b966d7697a8275d10b45eac"}
Jul 02 13:17:20.554269 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.554235    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f7cb747c-qh9k6_e0f775f1-3efe-4432-b60e-f184557e7f60/router/0.log"
Jul 02 13:17:20.557839 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.557813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:20.557980 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.557866    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:20.563446 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.563425    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:20.581045 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:20.581013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vm7c9_b1311dc9-756f-41d6-8749-bfdbd03fe4cd/serve-healthcheck-canary/0.log"
Jul 02 13:17:21.183247 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.183212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:21.237067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.237020    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:21.282753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.282098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdbs7" event={"ID":"eba184bd-079b-4132-9470-69224d9eed57","Type":"ContainerStarted","Data":"fea31b6d1432a3bac166a9608fbace0782e4ff2a96426122a2b441620140f0cb"}
Jul 02 13:17:21.282753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.282207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdbs7" event={"ID":"eba184bd-079b-4132-9470-69224d9eed57","Type":"ContainerStarted","Data":"48698492100dccb97abdf3e4d30621f75e69c09790edef89e1aa8a41a11b39b7"}
Jul 02 13:17:21.287407 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.287377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:21.304206 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.303955    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xdbs7" podStartSLOduration=1.8561241929999999 podStartE2EDuration="3.303919863s" totalImagesPullingTime="1.384685091s" totalInitContainerRuntime="63.110579ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:21.301054142 +0000 UTC m=+146.970861510" watchObservedRunningTime="2026-07-02 13:17:21.303919863 +0000 UTC m=+146.973727210"
Jul 02 13:17:21.392728 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.392694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:21.493205 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.493180    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:21.593579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:21.593550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:22.288069 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.288029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm" event={"ID":"d1e83c05-f5fc-4f05-92f3-0e41c86fc624","Type":"ContainerStarted","Data":"7f0819e96c8c878d244653835a2da62faa8464cce4ea44232dff106ef541bf80"}
Jul 02 13:17:22.288564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.288095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm" event={"ID":"d1e83c05-f5fc-4f05-92f3-0e41c86fc624","Type":"ContainerStarted","Data":"5c14765c390f8700cc80c1c0276069f7009c32377cce8922c48e182f1c1a2e2d"}
Jul 02 13:17:22.288564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.288119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm" event={"ID":"d1e83c05-f5fc-4f05-92f3-0e41c86fc624","Type":"ContainerStarted","Data":"6d0d9cbef27d21e6d63b4e61465a516eded82725671995fbc98085a2693ed441"}
Jul 02 13:17:22.290098 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.290071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn" event={"ID":"e88fec06-a219-4b2e-8e56-f71a07250ee1","Type":"ContainerStarted","Data":"f279deb99eab3c3b6030e589036b0b35932c73e156e597b7ee0bf034c111acf0"}
Jul 02 13:17:22.307742 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.307687    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-w7lzm" podStartSLOduration=2.197735034 podStartE2EDuration="4.307671933s" totalImagesPullingTime="2.109936899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:22.307052525 +0000 UTC m=+147.976859893" watchObservedRunningTime="2026-07-02 13:17:22.307671933 +0000 UTC m=+147.977479301"
Jul 02 13:17:22.324366 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:22.324312    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-2lrbn" podStartSLOduration=2.342405082 podStartE2EDuration="4.3242939s" totalImagesPullingTime="1.981888818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:22.323200494 +0000 UTC m=+147.993007863" watchObservedRunningTime="2026-07-02 13:17:22.3242939 +0000 UTC m=+147.994101269"
Jul 02 13:17:23.009140 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:23.009105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.080197 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:23.080163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.168168 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:23.168130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.319877 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:23.319841    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:24.038342 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:24.038302    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:24.094995 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:24.094955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:24.496405 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:24.496323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:30.158907 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:30.158869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:30.217220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:30.217185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:33.967095 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:33.967056    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:17:34.077988 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:34.077954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:34.133354 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:34.133317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:34.187956 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:34.187907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:39.035201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:39.035162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:39.134764 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:39.134724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:54.068079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:54.067995    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:54.124858 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:54.124830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:58.988380 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:58.988333    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5bc8df7d7b-sv9p8" podUID="f3db4f73-b762-4472-a2ee-359225bb9f99" containerName="console" containerID="cri-o://61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5" gracePeriod=15
Jul 02 13:17:59.233893 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.233869    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:59.239636 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn7gx\" (UniqueName: \"kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239640    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239795 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239704    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239847 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.239968 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.239919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca\") pod \"f3db4f73-b762-4472-a2ee-359225bb9f99\" (UID: \"f3db4f73-b762-4472-a2ee-359225bb9f99\") "
Jul 02 13:17:59.240119 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240095    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:59.240119 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240095    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config" (OuterVolumeSpecName: "console-config") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:59.240283 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240180    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:59.240283 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240191    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-trusted-ca-bundle\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.240283 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240236    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-console-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.240428 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.240405    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca" (OuterVolumeSpecName: "service-ca") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:59.241865 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.241834    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx" (OuterVolumeSpecName: "kube-api-access-qn7gx") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "kube-api-access-qn7gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:59.241971 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.241868    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:59.241971 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.241882    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f3db4f73-b762-4472-a2ee-359225bb9f99" (UID: "f3db4f73-b762-4472-a2ee-359225bb9f99"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:59.341003 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.340967    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-serving-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.341003 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.340996    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-oauth-serving-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.341003 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.341005    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3db4f73-b762-4472-a2ee-359225bb9f99-console-oauth-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.341234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.341016    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3db4f73-b762-4472-a2ee-359225bb9f99-service-ca\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.341234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.341025    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7gx\" (UniqueName: \"kubernetes.io/projected/f3db4f73-b762-4472-a2ee-359225bb9f99-kube-api-access-qn7gx\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:17:59.422145 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.422105    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3db4f73-b762-4472-a2ee-359225bb9f99" containerID="61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5" exitCode=0
Jul 02 13:17:59.422288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.422152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8df7d7b-sv9p8"
Jul 02 13:17:59.422288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.422175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8df7d7b-sv9p8" event={"ID":"f3db4f73-b762-4472-a2ee-359225bb9f99","Type":"ContainerDied","Data":"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"}
Jul 02 13:17:59.422288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.422217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8df7d7b-sv9p8" event={"ID":"f3db4f73-b762-4472-a2ee-359225bb9f99","Type":"ContainerDied","Data":"c6e44ce60cef86ba3775f51ad941cafc3f47e117ea3a9f499bf9936fa5294e6f"}
Jul 02 13:17:59.422288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.422236    2565 scope.go:117] "RemoveContainer" containerID="61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"
Jul 02 13:17:59.430580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.430565    2565 scope.go:117] "RemoveContainer" containerID="61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"
Jul 02 13:17:59.430825 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:17:59.430801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5\": container with ID starting with 61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5 not found: ID does not exist" containerID="61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"
Jul 02 13:17:59.430901 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.430833    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5"} err="failed to get container status \"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5\": rpc error: code = NotFound desc = could not find container \"61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5\": container with ID starting with 61c6a5b4a4dd3a52d83b0622e26d18d80978665cc7cad064f5b58f6d5e4025d5 not found: ID does not exist"
Jul 02 13:17:59.443612 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.443588    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:17:59.447303 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:17:59.447283    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc8df7d7b-sv9p8"]
Jul 02 13:18:00.706356 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:00.706320    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3db4f73-b762-4472-a2ee-359225bb9f99" path="/var/lib/kubelet/pods/f3db4f73-b762-4472-a2ee-359225bb9f99/volumes"
Jul 02 13:18:37.573386 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:37.573346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:37.885365 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:37.885281    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:47.003685 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:47.003641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:47.099580 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:47.099543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:50.779082 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:50.779033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:50.833032 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:50.832998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:52.991381 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:52.991348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:57.403730 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:57.403693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:57.505318 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:57.505285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:57.569163 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:18:57.569124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:52.815584 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:19:52.815526    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:52.874753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:19:52.874711    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:54.640466 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:19:54.640435    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:19:54.642651 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:19:54.642629    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:19:54.649764 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:19:54.649748    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:20:04.850204 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:04.850155    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:04.908873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:04.908833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:09.363159 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:09.363127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:09.463470 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:09.463437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:24.237310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:24.237256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:26.150850 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:26.150810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:34.796215 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:34.796177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:35.642680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:35.642651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:35.697917 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:20:35.697884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:34.803786 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:21:34.803748    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:18.299787 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:22:18.299745    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:02.591133 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:02.591084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:02.689693 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:02.689659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:02.934507 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:02.934429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:03.035111 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.035074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:03.135103 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.135072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:03.866565 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.866532    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"]
Jul 02 13:23:03.866995 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.866785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3db4f73-b762-4472-a2ee-359225bb9f99" containerName="console"
Jul 02 13:23:03.866995 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.866796    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3db4f73-b762-4472-a2ee-359225bb9f99" containerName="console"
Jul 02 13:23:03.866995 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.866842    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3db4f73-b762-4472-a2ee-359225bb9f99" containerName="console"
Jul 02 13:23:03.869707 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.869690    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:03.872148 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.872124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:23:03.873291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.873278    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:23:03.873390 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.873366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n8h28\""
Jul 02 13:23:03.881989 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.881963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"]
Jul 02 13:23:03.971440 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.971411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsp9j\" (UniqueName: \"kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:03.971585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.971474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:03.971585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:03.971511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.072835 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.072793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsp9j\" (UniqueName: \"kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.072998 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.072869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.072998 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.072907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.073261 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.073244    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.073300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.073276    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.082379 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.082361    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsp9j\" (UniqueName: \"kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.178673 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.178596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:04.303063 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.303029    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"]
Jul 02 13:23:04.305894 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:23:04.305872    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7d6ab0_cfe7_4e90_b86c_826ceebb385d.slice/crio-7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66 WatchSource:0}: Error finding container 7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66: Status 404 returned error can't find the container with id 7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66
Jul 02 13:23:04.307813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.307796    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:23:04.450221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:04.450156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2" event={"ID":"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d","Type":"ContainerStarted","Data":"7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66"}
Jul 02 13:23:09.470759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:09.470676    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerID="7418f10f0f4b1062bc4a95c735a4853f87c98101c38a993e0d51ed8676c74836" exitCode=0
Jul 02 13:23:09.471126 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:09.470763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2" event={"ID":"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d","Type":"ContainerDied","Data":"7418f10f0f4b1062bc4a95c735a4853f87c98101c38a993e0d51ed8676c74836"}
Jul 02 13:23:12.483693 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:12.483664    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerID="a7ac9787788a2657b816e67f12ea76a7330d7c1e27df0a3714500ff6e628260b" exitCode=0
Jul 02 13:23:12.484074 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:12.483728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2" event={"ID":"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d","Type":"ContainerDied","Data":"a7ac9787788a2657b816e67f12ea76a7330d7c1e27df0a3714500ff6e628260b"}
Jul 02 13:23:22.522101 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:22.522064    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerID="2df3600d18ebd3a362e3f1edf8ba08fa3edff57054f8afb902248e7a53d493c8" exitCode=0
Jul 02 13:23:22.522471 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:22.522122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2" event={"ID":"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d","Type":"ContainerDied","Data":"2df3600d18ebd3a362e3f1edf8ba08fa3edff57054f8afb902248e7a53d493c8"}
Jul 02 13:23:23.646843 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.646819    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:23.829891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.829813    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle\") pod \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") "
Jul 02 13:23:23.829891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.829869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsp9j\" (UniqueName: \"kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j\") pod \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") "
Jul 02 13:23:23.830128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.829916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util\") pod \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\" (UID: \"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d\") "
Jul 02 13:23:23.830643 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.830615    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle" (OuterVolumeSpecName: "bundle") pod "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" (UID: "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:23.832052 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.832030    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j" (OuterVolumeSpecName: "kube-api-access-hsp9j") pod "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" (UID: "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d"). InnerVolumeSpecName "kube-api-access-hsp9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:23.834655 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.834636    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util" (OuterVolumeSpecName: "util") pod "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" (UID: "5d7d6ab0-cfe7-4e90-b86c-826ceebb385d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:23.930642 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.930613    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsp9j\" (UniqueName: \"kubernetes.io/projected/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-kube-api-access-hsp9j\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:23:23.930642 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.930636    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-util\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:23:23.930642 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:23.930647    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d7d6ab0-cfe7-4e90-b86c-826ceebb385d-bundle\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:23:24.530709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:24.530672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2" event={"ID":"5d7d6ab0-cfe7-4e90-b86c-826ceebb385d","Type":"ContainerDied","Data":"7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66"}
Jul 02 13:23:24.530874 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:24.530720    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e134cccb892f7f5cb32a32cf2ddbf260cec59b69e61bc9f8af6ee91843e1b66"
Jul 02 13:23:24.530874 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:24.530766    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lwhw2"
Jul 02 13:23:29.892396 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:29.892332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:29.989325 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:29.989289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:35.612327 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:35.612293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:35.877053 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:35.876972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.087143 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.087102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.182152 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.182057    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.246540 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.246495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.321177 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.321133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.382421 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.382394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.644212 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.644180    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.844386 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.844352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:36.987775 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:36.987673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:37.086436 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:37.086408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:49.533842 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.533757    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78b97bdf87-nbnkk"]
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="util"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534048    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="util"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534059    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="extract"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534064    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="extract"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534076    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="pull"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534081    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="pull"
Jul 02 13:23:49.534232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.534129    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7d6ab0-cfe7-4e90-b86c-826ceebb385d" containerName="extract"
Jul 02 13:23:49.537214 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.537194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.539569 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.539545    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:23:49.542120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:23:49.542452 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542433    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:23:49.542533 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542503    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:23:49.542669 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542437    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:23:49.542848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:23:49.543654 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.542901    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:23:49.544009 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.543847    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-s7qpc\""
Jul 02 13:23:49.550397 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.549914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b97bdf87-nbnkk"]
Jul 02 13:23:49.550569 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.550544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:23:49.588687 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.588662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:49.596786 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqh9\" (UniqueName: \"kubernetes.io/projected/5c1fe43d-d733-4a6b-958d-840c0ab14535-kube-api-access-nrqh9\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.596883 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-trusted-ca-bundle\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.596883 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-service-ca\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.597007 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-oauth-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.597007 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.597007 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-oauth-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.597007 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.596997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.697638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.697785 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqh9\" (UniqueName: \"kubernetes.io/projected/5c1fe43d-d733-4a6b-958d-840c0ab14535-kube-api-access-nrqh9\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.697785 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-trusted-ca-bundle\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.697785 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-service-ca\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.697785 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-oauth-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698023 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698023 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.697978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-oauth-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698629 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.698598    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-service-ca\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698738 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.698648    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-oauth-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698738 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.698691    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.698738 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.698708    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c1fe43d-d733-4a6b-958d-840c0ab14535-trusted-ca-bundle\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.700164 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.700138    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-serving-cert\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.700249 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.700166    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c1fe43d-d733-4a6b-958d-840c0ab14535-console-oauth-config\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.711550 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.711531    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqh9\" (UniqueName: \"kubernetes.io/projected/5c1fe43d-d733-4a6b-958d-840c0ab14535-kube-api-access-nrqh9\") pod \"console-78b97bdf87-nbnkk\" (UID: \"5c1fe43d-d733-4a6b-958d-840c0ab14535\") " pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.854040 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.853959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:49.977530 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:49.977503    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b97bdf87-nbnkk"]
Jul 02 13:23:49.979704 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:23:49.979678    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1fe43d_d733_4a6b_958d_840c0ab14535.slice/crio-ec9fcd5e3dd964e4e2744f09e06a2306994ef20326960bb204b6b3fc463a7914 WatchSource:0}: Error finding container ec9fcd5e3dd964e4e2744f09e06a2306994ef20326960bb204b6b3fc463a7914: Status 404 returned error can't find the container with id ec9fcd5e3dd964e4e2744f09e06a2306994ef20326960bb204b6b3fc463a7914
Jul 02 13:23:50.626898 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:50.626862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b97bdf87-nbnkk" event={"ID":"5c1fe43d-d733-4a6b-958d-840c0ab14535","Type":"ContainerStarted","Data":"c924a13f3e97d218a72ff7b8150e9298fdf5b3ef94decca35a6d299309371365"}
Jul 02 13:23:50.627286 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:50.626915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b97bdf87-nbnkk" event={"ID":"5c1fe43d-d733-4a6b-958d-840c0ab14535","Type":"ContainerStarted","Data":"ec9fcd5e3dd964e4e2744f09e06a2306994ef20326960bb204b6b3fc463a7914"}
Jul 02 13:23:50.644025 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:50.643954    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78b97bdf87-nbnkk" podStartSLOduration=1.643923859 podStartE2EDuration="1.643923859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:50.642581456 +0000 UTC m=+536.312388825" watchObservedRunningTime="2026-07-02 13:23:50.643923859 +0000 UTC m=+536.313731226"
Jul 02 13:23:59.854088 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:59.854044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:59.854565 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:59.854108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:23:59.858539 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:23:59.858518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:24:00.666072 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:00.666045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78b97bdf87-nbnkk"
Jul 02 13:24:01.086125 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:01.086062    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:01.179289 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:01.179250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:15.868636 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:15.868604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:15.925644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:15.925600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:17.820914 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:17.820857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:17.920626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:17.920598    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:42.746501 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.746461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:42.809144 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.809112    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:24:42.812759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.812740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:42.815278 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.815254    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:24:42.815389 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.815292    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 13:24:42.816261 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.816245    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:24:42.816363 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.816262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hl2rz\""
Jul 02 13:24:42.817652 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.817628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:42.821679 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.821605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:24:42.845221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.845198    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-9f7lb"]
Jul 02 13:24:42.848210 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.848192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:42.850498 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.850479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6vpml\""
Jul 02 13:24:42.850498 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.850495    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 13:24:42.855714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.855696    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9f7lb"]
Jul 02 13:24:42.897536 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.897508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28gm\" (UniqueName: \"kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:42.897792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.897579    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:42.897792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.897757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:42.998254 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.998191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:42.998373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.998265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0882121-7575-4ae3-be86-e7762fba257c-data\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:42.998373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.998313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28gm\" (UniqueName: \"kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:42.998373 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:24:42.998319    2565 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 13:24:42.998373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:42.998352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtzn\" (UniqueName: \"kubernetes.io/projected/d0882121-7575-4ae3-be86-e7762fba257c-kube-api-access-jxtzn\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:42.998373 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:24:42.998374    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert podName:c8ed8811-e94f-4db3-9fe8-cffd4deecd88 nodeName:}" failed. No retries permitted until 2026-07-02 13:24:43.498359148 +0000 UTC m=+589.168166495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert") pod "kserve-controller-manager-694f5b7594-5trct" (UID: "c8ed8811-e94f-4db3-9fe8-cffd4deecd88") : secret "kserve-webhook-server-cert" not found
Jul 02 13:24:43.006854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.006835    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28gm\" (UniqueName: \"kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:43.099115 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.099085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtzn\" (UniqueName: \"kubernetes.io/projected/d0882121-7575-4ae3-be86-e7762fba257c-kube-api-access-jxtzn\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:43.099267 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.099170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0882121-7575-4ae3-be86-e7762fba257c-data\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:43.099481 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.099466    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0882121-7575-4ae3-be86-e7762fba257c-data\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:43.107881 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.107862    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtzn\" (UniqueName: \"kubernetes.io/projected/d0882121-7575-4ae3-be86-e7762fba257c-kube-api-access-jxtzn\") pod \"seaweedfs-64568bcd49-9f7lb\" (UID: \"d0882121-7575-4ae3-be86-e7762fba257c\") " pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:43.110902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.110865    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:43.157967 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.157928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:43.222201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.222136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:43.284361 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.284330    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9f7lb"]
Jul 02 13:24:43.287703 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:24:43.287673    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0882121_7575_4ae3_be86_e7762fba257c.slice/crio-6c65937fe74a1a424b646b5a2a96b847e0e1aad8bc82c2999bd5fc4dcf57d9b5 WatchSource:0}: Error finding container 6c65937fe74a1a424b646b5a2a96b847e0e1aad8bc82c2999bd5fc4dcf57d9b5: Status 404 returned error can't find the container with id 6c65937fe74a1a424b646b5a2a96b847e0e1aad8bc82c2999bd5fc4dcf57d9b5
Jul 02 13:24:43.288374 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.288346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:43.347121 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.347095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:43.502155 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.502128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:43.502285 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:24:43.502266    2565 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 13:24:43.502335 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:24:43.502326    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert podName:c8ed8811-e94f-4db3-9fe8-cffd4deecd88 nodeName:}" failed. No retries permitted until 2026-07-02 13:24:44.502311791 +0000 UTC m=+590.172119136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert") pod "kserve-controller-manager-694f5b7594-5trct" (UID: "c8ed8811-e94f-4db3-9fe8-cffd4deecd88") : secret "kserve-webhook-server-cert" not found
Jul 02 13:24:43.767824 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.767730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:43.816433 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:43.816369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9f7lb" event={"ID":"d0882121-7575-4ae3-be86-e7762fba257c","Type":"ContainerStarted","Data":"6c65937fe74a1a424b646b5a2a96b847e0e1aad8bc82c2999bd5fc4dcf57d9b5"}
Jul 02 13:24:44.512183 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:44.512147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:44.514873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:44.514839    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"kserve-controller-manager-694f5b7594-5trct\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") " pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:44.624509 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:44.624474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:44.760556 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:44.760535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:24:44.762952 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:24:44.762880    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ed8811_e94f_4db3_9fe8_cffd4deecd88.slice/crio-b86b02fde2a12600437261b5774dec722960d6d206a70da1be8e3621f2033043 WatchSource:0}: Error finding container b86b02fde2a12600437261b5774dec722960d6d206a70da1be8e3621f2033043: Status 404 returned error can't find the container with id b86b02fde2a12600437261b5774dec722960d6d206a70da1be8e3621f2033043
Jul 02 13:24:44.826780 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:44.826740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-5trct" event={"ID":"c8ed8811-e94f-4db3-9fe8-cffd4deecd88","Type":"ContainerStarted","Data":"b86b02fde2a12600437261b5774dec722960d6d206a70da1be8e3621f2033043"}
Jul 02 13:24:48.844622 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.844590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-5trct" event={"ID":"c8ed8811-e94f-4db3-9fe8-cffd4deecd88","Type":"ContainerStarted","Data":"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"}
Jul 02 13:24:48.845079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.844687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:24:48.846048 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.846028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9f7lb" event={"ID":"d0882121-7575-4ae3-be86-e7762fba257c","Type":"ContainerStarted","Data":"bf5bd3ee34fcbee3b8d6c7c3894fdd8ba2b1d0487ad4754d728265a889a6ab68"}
Jul 02 13:24:48.846170 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.846148    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:24:48.862264 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.862223    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-5trct" podStartSLOduration=3.4137122189999998 podStartE2EDuration="6.862210595s" totalImagesPullingTime="3.448498376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:48.860770705 +0000 UTC m=+594.530578074" watchObservedRunningTime="2026-07-02 13:24:48.862210595 +0000 UTC m=+594.532017962"
Jul 02 13:24:48.875604 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:48.875552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-9f7lb" podStartSLOduration=1.858129897 podStartE2EDuration="6.87553579s" totalImagesPullingTime="5.017405893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:48.875033387 +0000 UTC m=+594.544840755" watchObservedRunningTime="2026-07-02 13:24:48.87553579 +0000 UTC m=+594.545343158"
Jul 02 13:24:54.667388 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:54.667361    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:24:54.667786 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:54.667579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:24:54.851789 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:24:54.851759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-9f7lb"
Jul 02 13:25:04.764203 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:04.764170    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:19.689728 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.689694    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:25:19.692269 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.689900    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-694f5b7594-5trct" podUID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" containerName="manager" containerID="cri-o://8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037" gracePeriod=10
Jul 02 13:25:19.695634 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.695610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:25:19.712926 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.712905    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-4fd99"]
Jul 02 13:25:19.715973 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.715953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.727698 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.727671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-4fd99"]
Jul 02 13:25:19.791146 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.791118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8edec11c-29ad-4026-ab15-f1e3ca9af83d-cert\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.791267 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.791179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxb7d\" (UniqueName: \"kubernetes.io/projected/8edec11c-29ad-4026-ab15-f1e3ca9af83d-kube-api-access-wxb7d\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.891958 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.891906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8edec11c-29ad-4026-ab15-f1e3ca9af83d-cert\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.892113 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.892004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxb7d\" (UniqueName: \"kubernetes.io/projected/8edec11c-29ad-4026-ab15-f1e3ca9af83d-kube-api-access-wxb7d\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.894293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.894262    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8edec11c-29ad-4026-ab15-f1e3ca9af83d-cert\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.901406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.901381    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxb7d\" (UniqueName: \"kubernetes.io/projected/8edec11c-29ad-4026-ab15-f1e3ca9af83d-kube-api-access-wxb7d\") pod \"kserve-controller-manager-694f5b7594-4fd99\" (UID: \"8edec11c-29ad-4026-ab15-f1e3ca9af83d\") " pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:19.924546 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.924521    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:25:19.957963 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.957878    2565 generic.go:356] "Generic (PLEG): container finished" podID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" containerID="8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037" exitCode=0
Jul 02 13:25:19.957963 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.957924    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-5trct"
Jul 02 13:25:19.958101 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.957974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-5trct" event={"ID":"c8ed8811-e94f-4db3-9fe8-cffd4deecd88","Type":"ContainerDied","Data":"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"}
Jul 02 13:25:19.958101 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.958038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-5trct" event={"ID":"c8ed8811-e94f-4db3-9fe8-cffd4deecd88","Type":"ContainerDied","Data":"b86b02fde2a12600437261b5774dec722960d6d206a70da1be8e3621f2033043"}
Jul 02 13:25:19.958101 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.958070    2565 scope.go:117] "RemoveContainer" containerID="8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"
Jul 02 13:25:19.966035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.966011    2565 scope.go:117] "RemoveContainer" containerID="8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"
Jul 02 13:25:19.966305 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:25:19.966286    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037\": container with ID starting with 8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037 not found: ID does not exist" containerID="8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"
Jul 02 13:25:19.966396 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.966315    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037"} err="failed to get container status \"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037\": rpc error: code = NotFound desc = could not find container \"8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037\": container with ID starting with 8187d14de0824f2645d75d3533290c5c72aafb6f04378bade120fc163e760037 not found: ID does not exist"
Jul 02 13:25:19.992711 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.992688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") pod \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") "
Jul 02 13:25:19.992821 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.992723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w28gm\" (UniqueName: \"kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm\") pod \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\" (UID: \"c8ed8811-e94f-4db3-9fe8-cffd4deecd88\") "
Jul 02 13:25:19.995014 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.994919    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert" (OuterVolumeSpecName: "cert") pod "c8ed8811-e94f-4db3-9fe8-cffd4deecd88" (UID: "c8ed8811-e94f-4db3-9fe8-cffd4deecd88"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:25:19.995116 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:19.995007    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm" (OuterVolumeSpecName: "kube-api-access-w28gm") pod "c8ed8811-e94f-4db3-9fe8-cffd4deecd88" (UID: "c8ed8811-e94f-4db3-9fe8-cffd4deecd88"). InnerVolumeSpecName "kube-api-access-w28gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:25:20.074601 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.074549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:20.094143 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.094114    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:25:20.094250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.094148    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28gm\" (UniqueName: \"kubernetes.io/projected/c8ed8811-e94f-4db3-9fe8-cffd4deecd88-kube-api-access-w28gm\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:25:20.196169 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.196083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-4fd99"]
Jul 02 13:25:20.198696 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:25:20.198665    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edec11c_29ad_4026_ab15_f1e3ca9af83d.slice/crio-d51528b3b6627f0a294c439a942c5c7d22be7ea5efbdff4f306cfade342eb228 WatchSource:0}: Error finding container d51528b3b6627f0a294c439a942c5c7d22be7ea5efbdff4f306cfade342eb228: Status 404 returned error can't find the container with id d51528b3b6627f0a294c439a942c5c7d22be7ea5efbdff4f306cfade342eb228
Jul 02 13:25:20.281125 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.281099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:25:20.291111 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.291088    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-5trct"]
Jul 02 13:25:20.707831 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.707802    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" path="/var/lib/kubelet/pods/c8ed8811-e94f-4db3-9fe8-cffd4deecd88/volumes"
Jul 02 13:25:20.964311 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.964222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-4fd99" event={"ID":"8edec11c-29ad-4026-ab15-f1e3ca9af83d","Type":"ContainerStarted","Data":"980c8f4259fbe0835886d0e51122a2dc9e0cf9378fcd426a363a9d484da78179"}
Jul 02 13:25:20.964311 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.964277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-4fd99" event={"ID":"8edec11c-29ad-4026-ab15-f1e3ca9af83d","Type":"ContainerStarted","Data":"d51528b3b6627f0a294c439a942c5c7d22be7ea5efbdff4f306cfade342eb228"}
Jul 02 13:25:20.964518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.964352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:20.981964 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:20.981899    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-4fd99" podStartSLOduration=1.612283253 podStartE2EDuration="1.981883778s" totalImagesPullingTime="369.600525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:25:20.980503836 +0000 UTC m=+626.650311205" watchObservedRunningTime="2026-07-02 13:25:20.981883778 +0000 UTC m=+626.651691189"
Jul 02 13:25:31.660396 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:31.660359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:37.737201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:37.737131    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:51.972427 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:51.972397    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-4fd99"
Jul 02 13:25:52.963952 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:52.963856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:53.011211 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.011181    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xgsrx"]
Jul 02 13:25:53.011635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.011473    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" containerName="manager"
Jul 02 13:25:53.011635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.011485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" containerName="manager"
Jul 02 13:25:53.011635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.011550    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ed8811-e94f-4db3-9fe8-cffd4deecd88" containerName="manager"
Jul 02 13:25:53.014512 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.014496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.018379 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.018355    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 13:25:53.018502 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.018383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lsh75\""
Jul 02 13:25:53.028558 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.028533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xgsrx"]
Jul 02 13:25:53.072677 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.072628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:53.135772 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.135741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:53.156711 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.156679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plc6d\" (UniqueName: \"kubernetes.io/projected/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-kube-api-access-plc6d\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.156810 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.156754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-cert\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.257433 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.257356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-cert\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.257553 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.257436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plc6d\" (UniqueName: \"kubernetes.io/projected/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-kube-api-access-plc6d\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.259778 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.259758    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-cert\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.267371 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.267344    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plc6d\" (UniqueName: \"kubernetes.io/projected/0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f-kube-api-access-plc6d\") pod \"odh-model-controller-84b54b7f97-xgsrx\" (UID: \"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f\") " pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.325619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.325596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:53.382747 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.382661    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:53.448732 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.448667    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xgsrx"]
Jul 02 13:25:53.451031 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:25:53.451008    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad4d8b6_4a3e_4d6a_a0be_e92b2c4aec3f.slice/crio-8704a68d7ad3524238d10552cd8fd8d73187c86770e0480ae5516794e3f3006a WatchSource:0}: Error finding container 8704a68d7ad3524238d10552cd8fd8d73187c86770e0480ae5516794e3f3006a: Status 404 returned error can't find the container with id 8704a68d7ad3524238d10552cd8fd8d73187c86770e0480ae5516794e3f3006a
Jul 02 13:25:53.700591 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.700556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:53.800986 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:53.800957    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:54.092431 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:54.092394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xgsrx" event={"ID":"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f","Type":"ContainerStarted","Data":"8704a68d7ad3524238d10552cd8fd8d73187c86770e0480ae5516794e3f3006a"}
Jul 02 13:25:57.106312 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:57.106273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xgsrx" event={"ID":"0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f","Type":"ContainerStarted","Data":"b909c386261423c194c6ba9aae9d1bd7848b46a424bb3c797cda0b25b6e5a5da"}
Jul 02 13:25:57.106765 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:57.106410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:25:57.122983 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:25:57.122918    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-xgsrx" podStartSLOduration=2.000306533 podStartE2EDuration="5.122905448s" totalImagesPullingTime="3.122598915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:25:57.122815165 +0000 UTC m=+662.792622537" watchObservedRunningTime="2026-07-02 13:25:57.122905448 +0000 UTC m=+662.792712815"
Jul 02 13:26:08.112539 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.112502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-xgsrx"
Jul 02 13:26:08.965011 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.964978    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6k65s"]
Jul 02 13:26:08.969753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.969736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6k65s"
Jul 02 13:26:08.975432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.975395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnzp\" (UniqueName: \"kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:08.975635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.975615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:08.975789 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:08.975649    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6k65s"]
Jul 02 13:26:09.076227 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.076196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnzp\" (UniqueName: \"kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:09.076366 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.076246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:09.076586 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.076568    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:09.086525 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.086501    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnzp\" (UniqueName: \"kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp\") pod \"s3-init-6k65s\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") " pod="kserve/s3-init-6k65s"
Jul 02 13:26:09.287310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.287287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6k65s"
Jul 02 13:26:09.409487 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:09.409462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6k65s"]
Jul 02 13:26:09.411324 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:26:09.411293    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7eced54_ff41_4c47_bf44_e6c40c8b0fc4.slice/crio-0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f WatchSource:0}: Error finding container 0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f: Status 404 returned error can't find the container with id 0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f
Jul 02 13:26:10.161887 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:10.161856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerStarted","Data":"0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f"}
Jul 02 13:26:14.180917 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:14.180875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerStarted","Data":"bdd78bc0b5e44ceb1fa631d79d5bfbe8d8ec99768d39d055cdd2ecbe4c797fcc"}
Jul 02 13:26:15.186784 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:15.186750    2565 generic.go:356] "Generic (PLEG): container finished" podID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerID="bdd78bc0b5e44ceb1fa631d79d5bfbe8d8ec99768d39d055cdd2ecbe4c797fcc" exitCode=0
Jul 02 13:26:15.187150 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:15.186796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerDied","Data":"bdd78bc0b5e44ceb1fa631d79d5bfbe8d8ec99768d39d055cdd2ecbe4c797fcc"}
Jul 02 13:26:20.211543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:20.211507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerStarted","Data":"f09356e99633fb6c9d04c162ce590dfe1c6c85812616d7b8412ae9de8859056c"}
Jul 02 13:26:20.231665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:26:20.231612    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6k65s" podStartSLOduration=2.104817416 podStartE2EDuration="12.23159663s" totalImagesPullingTime="8.6046128s" totalInitContainerRuntime="1.522166414s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:20.230611743 +0000 UTC m=+685.900419112" watchObservedRunningTime="2026-07-02 13:26:20.23159663 +0000 UTC m=+685.901403997"
Jul 02 13:27:23.689636 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:23.689603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:25.468201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:25.468169    2565 generic.go:356] "Generic (PLEG): container finished" podID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerID="f09356e99633fb6c9d04c162ce590dfe1c6c85812616d7b8412ae9de8859056c" exitCode=0
Jul 02 13:27:25.468201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:25.468203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerDied","Data":"f09356e99633fb6c9d04c162ce590dfe1c6c85812616d7b8412ae9de8859056c"}
Jul 02 13:27:26.593585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.593556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6k65s"
Jul 02 13:27:26.662021 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.661997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtnzp\" (UniqueName: \"kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp\") pod \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") "
Jul 02 13:27:26.662159 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.662036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache\") pod \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\" (UID: \"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4\") "
Jul 02 13:27:26.664041 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.664017    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp" (OuterVolumeSpecName: "kube-api-access-wtnzp") pod "e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" (UID: "e7eced54-ff41-4c47-bf44-e6c40c8b0fc4"). InnerVolumeSpecName "kube-api-access-wtnzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:27:26.683545 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.683519    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" (UID: "e7eced54-ff41-4c47-bf44-e6c40c8b0fc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:26.763179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.763127    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnzp\" (UniqueName: \"kubernetes.io/projected/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-kube-api-access-wtnzp\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:27:26.763179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:26.763148    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eced54-ff41-4c47-bf44-e6c40c8b0fc4-model-cache\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:27:27.477889 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:27.477862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6k65s"
Jul 02 13:27:27.478070 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:27.477855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6k65s" event={"ID":"e7eced54-ff41-4c47-bf44-e6c40c8b0fc4","Type":"ContainerDied","Data":"0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f"}
Jul 02 13:27:27.478070 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:27.477983    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d3a7ee2bf6c0e441caff2db63be5b8d80ea850d00ef2a4c38a243b0d59a333f"
Jul 02 13:27:27.850411 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:27.850379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:27.950461 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:27.950429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:32.007177 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:32.007142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:39.287138 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.287101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:39.398870 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.398838    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:27:39.399174 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.399161    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerName="download-hf-model"
Jul 02 13:27:39.399225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.399176    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerName="download-hf-model"
Jul 02 13:27:39.399225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.399187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerName="s3-init"
Jul 02 13:27:39.399225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.399192    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerName="s3-init"
Jul 02 13:27:39.399319 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.399251    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7eced54-ff41-4c47-bf44-e6c40c8b0fc4" containerName="s3-init"
Jul 02 13:27:39.403488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.403470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.408224 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.408199    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dflzs\""
Jul 02 13:27:39.408350 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.408331    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d5968-predictor-serving-cert\""
Jul 02 13:27:39.408532 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.408511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:27:39.408696 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.408676    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:27:39.409005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.408987    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:27:39.409180 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.409160    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:27:39.409311 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.409291    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d5968-kube-rbac-proxy-sar-config\""
Jul 02 13:27:39.420187 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.420166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:27:39.456779 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.456754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.456923 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.456801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tgl\" (UniqueName: \"kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.456923 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.456826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.456923 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.456886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.457105 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.456963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.476028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.475995    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:39.557873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.557797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.557873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.557843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tgl\" (UniqueName: \"kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558077 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558126 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558165 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558490    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558811 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558790    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.558893 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.558798    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.560362 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.560339    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.567210 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.567190    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tgl\" (UniqueName: \"kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl\") pod \"isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.713923 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.713891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:27:39.851463 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:39.851440    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:27:39.853170 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:27:39.853144    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221cde2a_a18b_4c8e_8610_f73130e2c7eb.slice/crio-d5b629dbd243c6b0e5a2887054cfc3adb316ad547a7b8e17f370e2f27ae574c9 WatchSource:0}: Error finding container d5b629dbd243c6b0e5a2887054cfc3adb316ad547a7b8e17f370e2f27ae574c9: Status 404 returned error can't find the container with id d5b629dbd243c6b0e5a2887054cfc3adb316ad547a7b8e17f370e2f27ae574c9
Jul 02 13:27:40.529448 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:40.529412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerStarted","Data":"bff08f53522b55057a9ef271d5e10a7aff95efffae85be585083a820da4b3c8b"}
Jul 02 13:27:40.529821 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:40.529473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerStarted","Data":"d5b629dbd243c6b0e5a2887054cfc3adb316ad547a7b8e17f370e2f27ae574c9"}
Jul 02 13:27:43.227112 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:43.227081    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:44.547385 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:44.547348    2565 generic.go:356] "Generic (PLEG): container finished" podID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerID="bff08f53522b55057a9ef271d5e10a7aff95efffae85be585083a820da4b3c8b" exitCode=0
Jul 02 13:27:44.547748 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:44.547390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerDied","Data":"bff08f53522b55057a9ef271d5e10a7aff95efffae85be585083a820da4b3c8b"}
Jul 02 13:27:58.632702 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:27:58.632661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerStarted","Data":"cb48d641cfb643c9e65fee6592e0c8c98a81b555e2c9c89def0b05d674282317"}
Jul 02 13:28:00.642547 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:00.642516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerStarted","Data":"f367ce163396723cc10c2cfd944fbf52fbc4fd94144c5ecd57b8c9c2b0dab6a2"}
Jul 02 13:28:03.656948 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:03.656899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerStarted","Data":"82e070880a3d132a3efe637d52280b8ea851fa90034e7b9e527138d6d71da52a"}
Jul 02 13:28:03.657319 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:03.657096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:28:03.679521 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:03.679476    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podStartSLOduration=2.142117988 podStartE2EDuration="24.679462054s" totalImagesPullingTime="18.800413281s" totalInitContainerRuntime="3.736930785s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:27:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:28:03.678088287 +0000 UTC m=+789.347895656" watchObservedRunningTime="2026-07-02 13:28:03.679462054 +0000 UTC m=+789.349269421"
Jul 02 13:28:04.666099 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:04.666066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:28:04.666676 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:04.666658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:28:04.667689 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:04.667641    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:04.668751 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:04.668722    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:05.668775 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:05.668733    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:05.669246 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:05.669191    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:05.673213 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:05.673193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:28:06.672750 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:06.672709    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:06.673192 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:06.672979    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:07.676847 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:07.676804    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:07.677264 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:07.677126    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:17.677136 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:17.677086    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:17.677599 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:17.677574    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:27.676969 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:27.676913    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:27.677372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:27.677308    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:37.677373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:37.677317    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:37.677870 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:37.677843    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:47.676924 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:47.676873    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:47.677485 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:47.677461    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:28:57.677458 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:57.677412    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:28:57.677896 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:28:57.677783    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:07.676916 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:07.676867    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:29:07.677453 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:07.677341    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:17.677986 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:17.677955    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:17.678361 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:17.678054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:24.495891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.495854    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:24.540643 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.540612    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:29:24.541255 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.541221    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" containerID="cri-o://cb48d641cfb643c9e65fee6592e0c8c98a81b555e2c9c89def0b05d674282317" gracePeriod=30
Jul 02 13:29:24.541832 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.541804    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" containerID="cri-o://82e070880a3d132a3efe637d52280b8ea851fa90034e7b9e527138d6d71da52a" gracePeriod=30
Jul 02 13:29:24.541974 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.541914    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" containerID="cri-o://f367ce163396723cc10c2cfd944fbf52fbc4fd94144c5ecd57b8c9c2b0dab6a2" gracePeriod=30
Jul 02 13:29:24.543638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.543436    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:29:24.547681 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.547659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.570895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.564611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-17441-predictor-serving-cert\""
Jul 02 13:29:24.570895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.564676    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-17441-kube-rbac-proxy-sar-config\""
Jul 02 13:29:24.570895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.567783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:29:24.601889 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.601862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.602080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.601901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.602080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.601978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.602080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.602003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdtb\" (UniqueName: \"kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.602080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.602063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.603692 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.603668    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:24.617096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.617073    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:29:24.621015 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.621000    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.623209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.623191    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-17441-predictor-serving-cert\""
Jul 02 13:29:24.623403 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.623371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-17441-kube-rbac-proxy-sar-config\""
Jul 02 13:29:24.631665 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.631643    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:29:24.702638 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.702805 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.702805 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.702805 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.702805 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dkt\" (UniqueName: \"kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.703043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.703043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdtb\" (UniqueName: \"kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.702981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703532 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:24.703118    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-serving-cert: secret "isvc-sklearn-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:24.703532 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:24.703188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls podName:fc260412-6600-4d4a-8f15-d62092cf8500 nodeName:}" failed. No retries permitted until 2026-07-02 13:29:25.20316913 +0000 UTC m=+870.872976480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls") pod "isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500") : secret "isvc-sklearn-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:24.703646 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.703576    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703646 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.703610    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.703754 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.703665    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.712987 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.712929    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdtb\" (UniqueName: \"kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:24.804196 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dkt\" (UniqueName: \"kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.804372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.804372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.804372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.804372 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.804667 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:24.804492    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-serving-cert: secret "isvc-xgboost-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:24.804667 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:24.804559    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls podName:7ace76c1-97f1-4c88-9e63-6626cda974bf nodeName:}" failed. No retries permitted until 2026-07-02 13:29:25.304540143 +0000 UTC m=+870.974347494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls") pod "isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf") : secret "isvc-xgboost-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:24.804796 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.804688    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.805139 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.805102    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.805354 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.805336    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.812857 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.812833    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dkt\" (UniqueName: \"kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:24.970241 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.970205    2565 generic.go:356] "Generic (PLEG): container finished" podID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerID="f367ce163396723cc10c2cfd944fbf52fbc4fd94144c5ecd57b8c9c2b0dab6a2" exitCode=2
Jul 02 13:29:24.970417 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:24.970282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerDied","Data":"f367ce163396723cc10c2cfd944fbf52fbc4fd94144c5ecd57b8c9c2b0dab6a2"}
Jul 02 13:29:25.125571 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.125487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:25.206728 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.206688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:25.209093 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.209070    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") pod \"isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:25.307158 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.307122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:25.307319 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:25.307252    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-serving-cert: secret "isvc-xgboost-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:25.307358 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:29:25.307320    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls podName:7ace76c1-97f1-4c88-9e63-6626cda974bf nodeName:}" failed. No retries permitted until 2026-07-02 13:29:26.307306564 +0000 UTC m=+871.977113911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls") pod "isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf") : secret "isvc-xgboost-graph-raw-17441-predictor-serving-cert" not found
Jul 02 13:29:25.481169 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.481079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:25.625469 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.625446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:29:25.627816 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:29:25.627780    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc260412_6600_4d4a_8f15_d62092cf8500.slice/crio-892967d98d69cd2b6aef5c2cf42b18c647533ed70c5f15bf8860ff66efb09bad WatchSource:0}: Error finding container 892967d98d69cd2b6aef5c2cf42b18c647533ed70c5f15bf8860ff66efb09bad: Status 404 returned error can't find the container with id 892967d98d69cd2b6aef5c2cf42b18c647533ed70c5f15bf8860ff66efb09bad
Jul 02 13:29:25.668882 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.668856    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:25.788798 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.788765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:25.977635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.977596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerStarted","Data":"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"}
Jul 02 13:29:25.977825 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:25.977667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerStarted","Data":"892967d98d69cd2b6aef5c2cf42b18c647533ed70c5f15bf8860ff66efb09bad"}
Jul 02 13:29:26.315286 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.315252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:26.317654 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.317636    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:26.432624 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.432569    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:26.563795 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.563687    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:29:26.565955 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:29:26.565887    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ace76c1_97f1_4c88_9e63_6626cda974bf.slice/crio-7b1524ae46f3a511c74cbdfed435bbdb87d979d5c95d2e076d4130605ae92963 WatchSource:0}: Error finding container 7b1524ae46f3a511c74cbdfed435bbdb87d979d5c95d2e076d4130605ae92963: Status 404 returned error can't find the container with id 7b1524ae46f3a511c74cbdfed435bbdb87d979d5c95d2e076d4130605ae92963
Jul 02 13:29:26.983379 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.983297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerStarted","Data":"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"}
Jul 02 13:29:26.983815 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:26.983397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerStarted","Data":"7b1524ae46f3a511c74cbdfed435bbdb87d979d5c95d2e076d4130605ae92963"}
Jul 02 13:29:27.677108 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:27.677057    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:29:27.677428 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:27.677400    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:29.997506 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:29.997475    2565 generic.go:356] "Generic (PLEG): container finished" podID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerID="cb48d641cfb643c9e65fee6592e0c8c98a81b555e2c9c89def0b05d674282317" exitCode=0
Jul 02 13:29:29.997923 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:29.997546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerDied","Data":"cb48d641cfb643c9e65fee6592e0c8c98a81b555e2c9c89def0b05d674282317"}
Jul 02 13:29:29.998985 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:29.998963    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc260412-6600-4d4a-8f15-d62092cf8500" containerID="85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba" exitCode=0
Jul 02 13:29:29.999078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:29.999023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerDied","Data":"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"}
Jul 02 13:29:30.669515 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:30.669478    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:31.005106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.005028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerStarted","Data":"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"}
Jul 02 13:29:31.005106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.005093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerStarted","Data":"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"}
Jul 02 13:29:31.005568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.005393    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:31.005568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.005445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:31.006501 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.006473    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerID="9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36" exitCode=0
Jul 02 13:29:31.006631 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.006547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerDied","Data":"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"}
Jul 02 13:29:31.006631 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.006580    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:29:31.007777 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.007758    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:29:31.026170 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:31.026124    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podStartSLOduration=3.411120918 podStartE2EDuration="7.026110406s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.614989488s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:31.02373563 +0000 UTC m=+876.693543024" watchObservedRunningTime="2026-07-02 13:29:31.026110406 +0000 UTC m=+876.695917773"
Jul 02 13:29:32.013449 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:32.013406    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:29:35.669088 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:35.669037    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:35.669521 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:35.669259    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:37.020114 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:37.020082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:29:37.020724 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:37.020680    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:29:37.677152 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:37.677111    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:29:37.677498 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:37.677456    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:40.669895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:40.669850    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:45.669245 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:45.669198    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:47.021111 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:47.021015    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:29:47.677676 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:47.677613    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 13:29:47.677875 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:47.677808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:47.678019 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:47.677978    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:47.678115 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:47.678099    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:50.669897 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:50.669855    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 13:29:51.103200 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.103163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerStarted","Data":"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"}
Jul 02 13:29:51.103344 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.103231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerStarted","Data":"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"}
Jul 02 13:29:51.103520 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.103500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:51.103670 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.103657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:51.104634 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.104610    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:29:51.124711 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:51.124671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podStartSLOduration=3.784771338 podStartE2EDuration="27.124658408s" totalImagesPullingTime="19.759682145s" totalInitContainerRuntime="3.580204925s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:51.122220281 +0000 UTC m=+896.792027661" watchObservedRunningTime="2026-07-02 13:29:51.124658408 +0000 UTC m=+896.794465777"
Jul 02 13:29:52.108353 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:52.108308    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:29:54.696873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:54.696844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:29:54.697277 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:54.697157    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:29:55.123429 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.123382    2565 generic.go:356] "Generic (PLEG): container finished" podID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerID="82e070880a3d132a3efe637d52280b8ea851fa90034e7b9e527138d6d71da52a" exitCode=0
Jul 02 13:29:55.123548 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.123470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerDied","Data":"82e070880a3d132a3efe637d52280b8ea851fa90034e7b9e527138d6d71da52a"}
Jul 02 13:29:55.195852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.195830    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:55.274187 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config\") pod \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") "
Jul 02 13:29:55.274351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert\") pod \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") "
Jul 02 13:29:55.274351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location\") pod \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") "
Jul 02 13:29:55.274351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6tgl\" (UniqueName: \"kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl\") pod \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") "
Jul 02 13:29:55.274351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls\") pod \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\" (UID: \"221cde2a-a18b-4c8e-8610-f73130e2c7eb\") "
Jul 02 13:29:55.274629 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274597    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "221cde2a-a18b-4c8e-8610-f73130e2c7eb" (UID: "221cde2a-a18b-4c8e-8610-f73130e2c7eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:55.274684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274614    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "221cde2a-a18b-4c8e-8610-f73130e2c7eb" (UID: "221cde2a-a18b-4c8e-8610-f73130e2c7eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:55.274684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.274625    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "221cde2a-a18b-4c8e-8610-f73130e2c7eb" (UID: "221cde2a-a18b-4c8e-8610-f73130e2c7eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:55.276503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.276477    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "221cde2a-a18b-4c8e-8610-f73130e2c7eb" (UID: "221cde2a-a18b-4c8e-8610-f73130e2c7eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:55.276610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.276506    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl" (OuterVolumeSpecName: "kube-api-access-b6tgl") pod "221cde2a-a18b-4c8e-8610-f73130e2c7eb" (UID: "221cde2a-a18b-4c8e-8610-f73130e2c7eb"). InnerVolumeSpecName "kube-api-access-b6tgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:55.375664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.375606    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.375664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.375630    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.375664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.375641    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tgl\" (UniqueName: \"kubernetes.io/projected/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-api-access-b6tgl\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.375664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.375651    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/221cde2a-a18b-4c8e-8610-f73130e2c7eb-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:29:55.375664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:55.375660    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/221cde2a-a18b-4c8e-8610-f73130e2c7eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:29:56.130119 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.130084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr" event={"ID":"221cde2a-a18b-4c8e-8610-f73130e2c7eb","Type":"ContainerDied","Data":"d5b629dbd243c6b0e5a2887054cfc3adb316ad547a7b8e17f370e2f27ae574c9"}
Jul 02 13:29:56.130119 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.130107    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"
Jul 02 13:29:56.130561 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.130158    2565 scope.go:117] "RemoveContainer" containerID="82e070880a3d132a3efe637d52280b8ea851fa90034e7b9e527138d6d71da52a"
Jul 02 13:29:56.138507 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.138490    2565 scope.go:117] "RemoveContainer" containerID="f367ce163396723cc10c2cfd944fbf52fbc4fd94144c5ecd57b8c9c2b0dab6a2"
Jul 02 13:29:56.145881 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.145865    2565 scope.go:117] "RemoveContainer" containerID="cb48d641cfb643c9e65fee6592e0c8c98a81b555e2c9c89def0b05d674282317"
Jul 02 13:29:56.152612 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.152595    2565 scope.go:117] "RemoveContainer" containerID="bff08f53522b55057a9ef271d5e10a7aff95efffae85be585083a820da4b3c8b"
Jul 02 13:29:56.161718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.161694    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:29:56.165677 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.165655    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d5968-predictor-64b9f69486-zxzwr"]
Jul 02 13:29:56.707477 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:56.707448    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" path="/var/lib/kubelet/pods/221cde2a-a18b-4c8e-8610-f73130e2c7eb/volumes"
Jul 02 13:29:57.021516 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:57.021433    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:29:57.112570 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:57.112543    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:29:57.113067 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:29:57.113038    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:07.021017 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:07.020978    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:30:07.113150 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:07.113113    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:17.021619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:17.021579    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:30:17.113163 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:17.113127    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:27.021009 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:27.020972    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:30:27.113788 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:27.113747    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:37.021210 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:37.021166    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:30:37.113702 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:37.113665    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:41.702810 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:41.702764    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 13:30:47.113902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:47.113862    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:30:51.704073 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:51.704038    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:30:57.113732 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:30:57.113701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:31:04.587365 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:04.587329    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:05.986886 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:05.986851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:07.905962 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:07.905911    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:07.962347 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:07.962318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:14.640554 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.640515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:14.850240 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850208    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:31:14.850801 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850775    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy"
Jul 02 13:31:14.850801 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850801    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850824    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850832    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850845    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850851    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850861    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="storage-initializer"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850867    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="storage-initializer"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850956    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kube-rbac-proxy"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850972    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="agent"
Jul 02 13:31:14.851028 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.850980    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="221cde2a-a18b-4c8e-8610-f73130e2c7eb" containerName="kserve-container"
Jul 02 13:31:14.855458 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.855440    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.857572 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.857552    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2456a-predictor-serving-cert\""
Jul 02 13:31:14.857850 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.857830    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2456a-kube-rbac-proxy-sar-config\""
Jul 02 13:31:14.868382 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.868360    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:31:14.876795 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.876774    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:14.890648 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.890593    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:31:14.890958 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.890788    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container" containerID="cri-o://ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036" gracePeriod=30
Jul 02 13:31:14.890958 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.890817    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kube-rbac-proxy" containerID="cri-o://c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef" gracePeriod=30
Jul 02 13:31:14.906585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.906505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.906585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.906561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.906793 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.906611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.906793 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.906704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcd9\" (UniqueName: \"kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.906793 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.906751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:14.950630 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.950600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:14.956711 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.956660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:31:14.960242 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.960228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:14.962503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.962470    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2456a-kube-rbac-proxy-sar-config\""
Jul 02 13:31:14.962503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.962479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2456a-predictor-serving-cert\""
Jul 02 13:31:14.971931 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:14.971896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:31:15.006377 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.006350    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:31:15.006754 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.006600    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" containerID="cri-o://84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32" gracePeriod=30
Jul 02 13:31:15.006754 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.006630    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kube-rbac-proxy" containerID="cri-o://4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387" gracePeriod=30
Jul 02 13:31:15.007185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.007185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcd9\" (UniqueName: \"kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.007310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2vf\" (UniqueName: \"kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.007310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.007310 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.007474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.007474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.007474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.007474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.007680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.007488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.007680 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:15.007623    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-2456a-predictor-serving-cert" not found
Jul 02 13:31:15.007783 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:15.007708    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls podName:eb27b97f-5640-4afa-bd00-de5fdd3ea1db nodeName:}" failed. No retries permitted until 2026-07-02 13:31:15.507687635 +0000 UTC m=+981.177494980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db") : secret "isvc-sklearn-graph-raw-hpa-2456a-predictor-serving-cert" not found
Jul 02 13:31:15.008084 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.008060    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.008703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.008680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.008951 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.008904    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.016758 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.016729    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcd9\" (UniqueName: \"kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.020776 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.020748    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:15.108709 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.108683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.108854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.108726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2vf\" (UniqueName: \"kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.108854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.108752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.108854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.108813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.109103 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.108884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.109530 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.109508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.109589 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.109500    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.109731 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:15.109713    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-2456a-predictor-serving-cert" not found
Jul 02 13:31:15.109808 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:15.109796    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls podName:22b44a3a-d286-4537-b959-b90727a4eace nodeName:}" failed. No retries permitted until 2026-07-02 13:31:15.609773191 +0000 UTC m=+981.279580553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" (UID: "22b44a3a-d286-4537-b959-b90727a4eace") : secret "isvc-xgboost-graph-raw-hpa-2456a-predictor-serving-cert" not found
Jul 02 13:31:15.110064 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.110047    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.118507 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.118482    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2vf\" (UniqueName: \"kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.226810 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.226727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:15.363347 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.363251    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:15.451694 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.451662    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc260412-6600-4d4a-8f15-d62092cf8500" containerID="c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef" exitCode=2
Jul 02 13:31:15.451838 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.451739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerDied","Data":"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"}
Jul 02 13:31:15.453442 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.453423    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerID="4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387" exitCode=2
Jul 02 13:31:15.453522 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.453452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerDied","Data":"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"}
Jul 02 13:31:15.513238 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.513177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.515626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.515601    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.613984 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.613953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.616180 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.616154    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.765846 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.765775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:15.876293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.876262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:15.892698 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:31:15.892659    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb27b97f_5640_4afa_bd00_de5fdd3ea1db.slice/crio-52a83d703979e2a21c5d3bd9d7293a5ce168bbe8f659e328d01e7f001f32e8ee WatchSource:0}: Error finding container 52a83d703979e2a21c5d3bd9d7293a5ce168bbe8f659e328d01e7f001f32e8ee: Status 404 returned error can't find the container with id 52a83d703979e2a21c5d3bd9d7293a5ce168bbe8f659e328d01e7f001f32e8ee
Jul 02 13:31:15.893018 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:15.892995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:31:16.012763 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:16.012733    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:31:16.015065 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:31:16.015037    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b44a3a_d286_4537_b959_b90727a4eace.slice/crio-112661d082da7558bb9bf333b585d8a4c0d1caefb5dbf27e230c7284c6568eb9 WatchSource:0}: Error finding container 112661d082da7558bb9bf333b585d8a4c0d1caefb5dbf27e230c7284c6568eb9: Status 404 returned error can't find the container with id 112661d082da7558bb9bf333b585d8a4c0d1caefb5dbf27e230c7284c6568eb9
Jul 02 13:31:16.470864 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:16.470822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerStarted","Data":"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"}
Jul 02 13:31:16.471033 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:16.470927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerStarted","Data":"112661d082da7558bb9bf333b585d8a4c0d1caefb5dbf27e230c7284c6568eb9"}
Jul 02 13:31:16.472378 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:16.472353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerStarted","Data":"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"}
Jul 02 13:31:16.472464 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:16.472399    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerStarted","Data":"52a83d703979e2a21c5d3bd9d7293a5ce168bbe8f659e328d01e7f001f32e8ee"}
Jul 02 13:31:17.013593 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:17.013556    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 02 13:31:17.109427 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:17.109387    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:31:17.113769 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:17.113743    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:31:18.660045 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.660014    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:31:18.739472 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739402    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location\") pod \"7ace76c1-97f1-4c88-9e63-6626cda974bf\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") "
Jul 02 13:31:18.739472 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config\") pod \"7ace76c1-97f1-4c88-9e63-6626cda974bf\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") "
Jul 02 13:31:18.739472 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") pod \"7ace76c1-97f1-4c88-9e63-6626cda974bf\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") "
Jul 02 13:31:18.739731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739503    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert\") pod \"7ace76c1-97f1-4c88-9e63-6626cda974bf\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") "
Jul 02 13:31:18.739731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739552    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4dkt\" (UniqueName: \"kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt\") pod \"7ace76c1-97f1-4c88-9e63-6626cda974bf\" (UID: \"7ace76c1-97f1-4c88-9e63-6626cda974bf\") "
Jul 02 13:31:18.739731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739714    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ace76c1-97f1-4c88-9e63-6626cda974bf" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:31:18.739872 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739848    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ace76c1-97f1-4c88-9e63-6626cda974bf-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:18.739872 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739850    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ace76c1-97f1-4c88-9e63-6626cda974bf" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:31:18.739967 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.739896    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ace76c1-97f1-4c88-9e63-6626cda974bf" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:31:18.741610 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.741590    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ace76c1-97f1-4c88-9e63-6626cda974bf" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:31:18.741684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.741592    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt" (OuterVolumeSpecName: "kube-api-access-d4dkt") pod "7ace76c1-97f1-4c88-9e63-6626cda974bf" (UID: "7ace76c1-97f1-4c88-9e63-6626cda974bf"). InnerVolumeSpecName "kube-api-access-d4dkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:31:18.840301 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.840275    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:18.840301 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.840298    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ace76c1-97f1-4c88-9e63-6626cda974bf-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:18.840451 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.840308    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ace76c1-97f1-4c88-9e63-6626cda974bf-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:18.840451 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:18.840317    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dkt\" (UniqueName: \"kubernetes.io/projected/7ace76c1-97f1-4c88-9e63-6626cda974bf-kube-api-access-d4dkt\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:19.486452 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.486424    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerID="84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32" exitCode=0
Jul 02 13:31:19.486543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.486484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerDied","Data":"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"}
Jul 02 13:31:19.486543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.486488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"
Jul 02 13:31:19.486626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.486554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp" event={"ID":"7ace76c1-97f1-4c88-9e63-6626cda974bf","Type":"ContainerDied","Data":"7b1524ae46f3a511c74cbdfed435bbdb87d979d5c95d2e076d4130605ae92963"}
Jul 02 13:31:19.486626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.486579    2565 scope.go:117] "RemoveContainer" containerID="4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"
Jul 02 13:31:19.531856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.531769    2565 scope.go:117] "RemoveContainer" containerID="84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"
Jul 02 13:31:19.540190 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.540173    2565 scope.go:117] "RemoveContainer" containerID="9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"
Jul 02 13:31:19.548020 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.547997    2565 scope.go:117] "RemoveContainer" containerID="4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"
Jul 02 13:31:19.548303 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:19.548282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387\": container with ID starting with 4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387 not found: ID does not exist" containerID="4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"
Jul 02 13:31:19.548402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.548316    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387"} err="failed to get container status \"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387\": rpc error: code = NotFound desc = could not find container \"4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387\": container with ID starting with 4cefcf3df199bcd2ca9f1dcf2dbbe412954245024da4e79f7146787d475c4387 not found: ID does not exist"
Jul 02 13:31:19.548402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.548342    2565 scope.go:117] "RemoveContainer" containerID="84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"
Jul 02 13:31:19.548599 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:19.548572    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32\": container with ID starting with 84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32 not found: ID does not exist" containerID="84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"
Jul 02 13:31:19.548644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.548607    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32"} err="failed to get container status \"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32\": rpc error: code = NotFound desc = could not find container \"84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32\": container with ID starting with 84a2d2e54490025f5dfa47e1c86fe256d37587808584c79d66fabe53c691de32 not found: ID does not exist"
Jul 02 13:31:19.548644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.548630    2565 scope.go:117] "RemoveContainer" containerID="9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"
Jul 02 13:31:19.548973 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:19.548926    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36\": container with ID starting with 9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36 not found: ID does not exist" containerID="9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"
Jul 02 13:31:19.549065 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.548974    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36"} err="failed to get container status \"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36\": rpc error: code = NotFound desc = could not find container \"9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36\": container with ID starting with 9112b9b0ca2b034a4fce3caee404380a00bedf2c3b95349d63761f2f3ec8eb36 not found: ID does not exist"
Jul 02 13:31:19.549269 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.549248    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:31:19.561345 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.561320    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-17441-predictor-7d56dbf8d9-zsmzp"]
Jul 02 13:31:19.653132 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.653112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:31:19.746714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.746646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") pod \"fc260412-6600-4d4a-8f15-d62092cf8500\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") "
Jul 02 13:31:19.746714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.746688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config\") pod \"fc260412-6600-4d4a-8f15-d62092cf8500\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") "
Jul 02 13:31:19.747185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.746740    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert\") pod \"fc260412-6600-4d4a-8f15-d62092cf8500\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") "
Jul 02 13:31:19.747185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.746760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdtb\" (UniqueName: \"kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb\") pod \"fc260412-6600-4d4a-8f15-d62092cf8500\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") "
Jul 02 13:31:19.747185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.746787    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location\") pod \"fc260412-6600-4d4a-8f15-d62092cf8500\" (UID: \"fc260412-6600-4d4a-8f15-d62092cf8500\") "
Jul 02 13:31:19.747185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.747134    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc260412-6600-4d4a-8f15-d62092cf8500" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:31:19.747346 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.747186    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc260412-6600-4d4a-8f15-d62092cf8500" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:31:19.747346 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.747217    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc260412-6600-4d4a-8f15-d62092cf8500" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:31:19.748881 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.748859    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb" (OuterVolumeSpecName: "kube-api-access-qmdtb") pod "fc260412-6600-4d4a-8f15-d62092cf8500" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500"). InnerVolumeSpecName "kube-api-access-qmdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:31:19.748956 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.748867    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc260412-6600-4d4a-8f15-d62092cf8500" (UID: "fc260412-6600-4d4a-8f15-d62092cf8500"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:31:19.847882 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.847855    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc260412-6600-4d4a-8f15-d62092cf8500-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:19.847882 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.847880    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:19.848050 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.847895    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc260412-6600-4d4a-8f15-d62092cf8500-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:19.848050 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.847907    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdtb\" (UniqueName: \"kubernetes.io/projected/fc260412-6600-4d4a-8f15-d62092cf8500-kube-api-access-qmdtb\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:19.848050 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:19.847920    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc260412-6600-4d4a-8f15-d62092cf8500-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:31:20.492438 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.492404    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc260412-6600-4d4a-8f15-d62092cf8500" containerID="ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036" exitCode=0
Jul 02 13:31:20.492632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.492468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"
Jul 02 13:31:20.492632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.492484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerDied","Data":"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"}
Jul 02 13:31:20.492632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.492562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt" event={"ID":"fc260412-6600-4d4a-8f15-d62092cf8500","Type":"ContainerDied","Data":"892967d98d69cd2b6aef5c2cf42b18c647533ed70c5f15bf8860ff66efb09bad"}
Jul 02 13:31:20.492632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.492595    2565 scope.go:117] "RemoveContainer" containerID="c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"
Jul 02 13:31:20.494905 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.494879    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerID="5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d" exitCode=0
Jul 02 13:31:20.495024 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.494921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerDied","Data":"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"}
Jul 02 13:31:20.496651 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.496626    2565 generic.go:356] "Generic (PLEG): container finished" podID="22b44a3a-d286-4537-b959-b90727a4eace" containerID="464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2" exitCode=0
Jul 02 13:31:20.496749 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.496662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerDied","Data":"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"}
Jul 02 13:31:20.502062 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.502040    2565 scope.go:117] "RemoveContainer" containerID="ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"
Jul 02 13:31:20.509841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.509821    2565 scope.go:117] "RemoveContainer" containerID="85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"
Jul 02 13:31:20.527900 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.527882    2565 scope.go:117] "RemoveContainer" containerID="c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"
Jul 02 13:31:20.528160 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:20.528141    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef\": container with ID starting with c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef not found: ID does not exist" containerID="c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"
Jul 02 13:31:20.528234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.528172    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef"} err="failed to get container status \"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef\": rpc error: code = NotFound desc = could not find container \"c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef\": container with ID starting with c0ea41a7ee98c874d9a4dc9bbf8fec91556bec008f64075389fcec60493163ef not found: ID does not exist"
Jul 02 13:31:20.528234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.528210    2565 scope.go:117] "RemoveContainer" containerID="ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"
Jul 02 13:31:20.528458 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:20.528442    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036\": container with ID starting with ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036 not found: ID does not exist" containerID="ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"
Jul 02 13:31:20.528504 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.528464    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036"} err="failed to get container status \"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036\": rpc error: code = NotFound desc = could not find container \"ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036\": container with ID starting with ec56e7d2d1b944c27db9c2537168ffeeb4df47b0688ec0db73324a1f2edb5036 not found: ID does not exist"
Jul 02 13:31:20.528504 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.528483    2565 scope.go:117] "RemoveContainer" containerID="85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"
Jul 02 13:31:20.528689 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:31:20.528672    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba\": container with ID starting with 85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba not found: ID does not exist" containerID="85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"
Jul 02 13:31:20.528741 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.528697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba"} err="failed to get container status \"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba\": rpc error: code = NotFound desc = could not find container \"85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba\": container with ID starting with 85b67a7a3bfdd8f279b13fdfb6fb708d03acc8a48206e50ae22eb1c4fa23d4ba not found: ID does not exist"
Jul 02 13:31:20.558579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.558557    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:31:20.562843 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.562825    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-17441-predictor-6bbcdfff74-2npzt"]
Jul 02 13:31:20.707231 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.707196    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" path="/var/lib/kubelet/pods/7ace76c1-97f1-4c88-9e63-6626cda974bf/volumes"
Jul 02 13:31:20.707987 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:20.707966    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" path="/var/lib/kubelet/pods/fc260412-6600-4d4a-8f15-d62092cf8500/volumes"
Jul 02 13:31:21.503753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.503717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerStarted","Data":"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"}
Jul 02 13:31:21.504221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.503785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerStarted","Data":"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"}
Jul 02 13:31:21.504221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.504060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:21.504221 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.504213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:21.505356 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.505328    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:21.506625 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.506604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerStarted","Data":"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"}
Jul 02 13:31:21.506748 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.506653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerStarted","Data":"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"}
Jul 02 13:31:21.506891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.506877    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:21.506970 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.506920    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:21.507861 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.507840    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:31:21.526106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.526063    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podStartSLOduration=4.0031161 podStartE2EDuration="7.5260512s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.5229351s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:31:21.524224563 +0000 UTC m=+987.194031938" watchObservedRunningTime="2026-07-02 13:31:21.5260512 +0000 UTC m=+987.195858568"
Jul 02 13:31:21.543100 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:21.543055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podStartSLOduration=3.850836389 podStartE2EDuration="7.54303973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.692203341s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:31:21.542546132 +0000 UTC m=+987.212353499" watchObservedRunningTime="2026-07-02 13:31:21.54303973 +0000 UTC m=+987.212847111"
Jul 02 13:31:22.513816 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:22.513773    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:31:22.514225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:22.513912    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:27.518885 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:27.518855    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:31:27.519328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:27.518962    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:31:27.519451 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:27.519428    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:27.519841 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:27.519818    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:31:37.520207 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:37.520161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:37.520567 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:37.520161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:31:47.520274 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:47.520233    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:47.520614 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:47.520233    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:31:57.520225 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:57.520192    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:31:57.520664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:31:57.520189    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:32:07.519462 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:07.519415    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:32:07.519918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:07.519647    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:32:17.520413 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:17.520367    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:32:17.520413 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:17.520397    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:32:27.520186 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:27.520138    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:32:27.520703 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:27.520686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:32:33.124879 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:33.124840    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:37.520104 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:37.520076    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:32:54.882143 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:54.881976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:55.149034 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:32:55.148913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:04.950264 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:04.950208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:05.097683 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.097647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:05.132035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132003    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:33:05.132322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132310    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132323    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132334    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kube-rbac-proxy"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132340    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kube-rbac-proxy"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132348    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kube-rbac-proxy"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132354    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kube-rbac-proxy"
Jul 02 13:33:05.132364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132362    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="storage-initializer"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132369    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="storage-initializer"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132377    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132390    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="storage-initializer"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132395    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="storage-initializer"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132441    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kserve-container"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132450    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kserve-container"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132457    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ace76c1-97f1-4c88-9e63-6626cda974bf" containerName="kube-rbac-proxy"
Jul 02 13:33:05.132644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.132462    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc260412-6600-4d4a-8f15-d62092cf8500" containerName="kube-rbac-proxy"
Jul 02 13:33:05.135732 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.135706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.139574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.139367    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-260d5-kube-rbac-proxy-sar-config\""
Jul 02 13:33:05.139574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.139378    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-260d5-predictor-serving-cert\""
Jul 02 13:33:05.149110 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.149071    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:33:05.170388 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.170357    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:05.174666 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.174643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:33:05.174875 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.174851    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" containerID="cri-o://8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78" gracePeriod=30
Jul 02 13:33:05.174967 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.174882    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kube-rbac-proxy" containerID="cri-o://dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38" gracePeriod=30
Jul 02 13:33:05.211664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.211604    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:33:05.211854 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.211824    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" containerID="cri-o://4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534" gracePeriod=30
Jul 02 13:33:05.212002 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.211867    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kube-rbac-proxy" containerID="cri-o://ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078" gracePeriod=30
Jul 02 13:33:05.234829 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.234802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:05.280771 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.280737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwql\" (UniqueName: \"kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.280886 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.280806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.280886 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.280849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.381218 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.381191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.381328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.381233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.381328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.381285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwql\" (UniqueName: \"kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.381819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.381801    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.383406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.383388    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.390490 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.390462    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwql\" (UniqueName: \"kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql\") pod \"message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") " pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.449894 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.449870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:05.574448 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.574424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:33:05.576241 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:33:05.576219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda986669e_f600_413f_bcf4_132fbcb39dd2.slice/crio-cd8382693732b344954fbb6c7b54166a346d31082725a3819ec80b06508ac284 WatchSource:0}: Error finding container cd8382693732b344954fbb6c7b54166a346d31082725a3819ec80b06508ac284: Status 404 returned error can't find the container with id cd8382693732b344954fbb6c7b54166a346d31082725a3819ec80b06508ac284
Jul 02 13:33:05.932195 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.932158    2565 generic.go:356] "Generic (PLEG): container finished" podID="22b44a3a-d286-4537-b959-b90727a4eace" containerID="ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078" exitCode=2
Jul 02 13:33:05.932364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.932229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerDied","Data":"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"}
Jul 02 13:33:05.934207 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.934187    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerID="dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38" exitCode=2
Jul 02 13:33:05.934312 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.934259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerDied","Data":"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"}
Jul 02 13:33:05.935191 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:05.935171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerStarted","Data":"cd8382693732b344954fbb6c7b54166a346d31082725a3819ec80b06508ac284"}
Jul 02 13:33:06.939988 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:06.939965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerStarted","Data":"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"}
Jul 02 13:33:07.514171 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.514133    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 02 13:33:07.514318 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.514137    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 02 13:33:07.519482 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.519460    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:33:07.519634 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.519618    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:33:07.945863 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.945829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerStarted","Data":"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"}
Jul 02 13:33:07.946337 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.945899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:07.946337 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.945929    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:07.947899 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.947878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:07.964136 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:07.964089    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" podStartSLOduration=1.693532196 podStartE2EDuration="2.964071493s" totalImagesPullingTime="1.270539297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:33:07.96235184 +0000 UTC m=+1093.632159208" watchObservedRunningTime="2026-07-02 13:33:07.964071493 +0000 UTC m=+1093.633878864"
Jul 02 13:33:08.754702 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.754677    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:33:08.908714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.908634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph2vf\" (UniqueName: \"kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf\") pod \"22b44a3a-d286-4537-b959-b90727a4eace\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") "
Jul 02 13:33:08.908714 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.908690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location\") pod \"22b44a3a-d286-4537-b959-b90727a4eace\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") "
Jul 02 13:33:08.909018 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.908722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config\") pod \"22b44a3a-d286-4537-b959-b90727a4eace\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") "
Jul 02 13:33:08.909018 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.908875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") pod \"22b44a3a-d286-4537-b959-b90727a4eace\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") "
Jul 02 13:33:08.909018 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.908955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert\") pod \"22b44a3a-d286-4537-b959-b90727a4eace\" (UID: \"22b44a3a-d286-4537-b959-b90727a4eace\") "
Jul 02 13:33:08.909154 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.909017    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22b44a3a-d286-4537-b959-b90727a4eace" (UID: "22b44a3a-d286-4537-b959-b90727a4eace"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:08.909154 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.909047    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22b44a3a-d286-4537-b959-b90727a4eace" (UID: "22b44a3a-d286-4537-b959-b90727a4eace"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:33:08.909263 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.909248    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:08.909297 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.909265    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b44a3a-d286-4537-b959-b90727a4eace-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:08.909297 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.909279    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22b44a3a-d286-4537-b959-b90727a4eace" (UID: "22b44a3a-d286-4537-b959-b90727a4eace"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:33:08.910781 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.910760    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf" (OuterVolumeSpecName: "kube-api-access-ph2vf") pod "22b44a3a-d286-4537-b959-b90727a4eace" (UID: "22b44a3a-d286-4537-b959-b90727a4eace"). InnerVolumeSpecName "kube-api-access-ph2vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:33:08.910869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.910848    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22b44a3a-d286-4537-b959-b90727a4eace" (UID: "22b44a3a-d286-4537-b959-b90727a4eace"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:33:08.951389 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.951361    2565 generic.go:356] "Generic (PLEG): container finished" podID="22b44a3a-d286-4537-b959-b90727a4eace" containerID="4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534" exitCode=0
Jul 02 13:33:08.951718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.951423    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"
Jul 02 13:33:08.951718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.951435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerDied","Data":"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"}
Jul 02 13:33:08.951718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.951503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh" event={"ID":"22b44a3a-d286-4537-b959-b90727a4eace","Type":"ContainerDied","Data":"112661d082da7558bb9bf333b585d8a4c0d1caefb5dbf27e230c7284c6568eb9"}
Jul 02 13:33:08.951718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.951534    2565 scope.go:117] "RemoveContainer" containerID="ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"
Jul 02 13:33:08.960197 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.960077    2565 scope.go:117] "RemoveContainer" containerID="4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"
Jul 02 13:33:08.967635 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.967615    2565 scope.go:117] "RemoveContainer" containerID="464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"
Jul 02 13:33:08.975285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975266    2565 scope.go:117] "RemoveContainer" containerID="ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"
Jul 02 13:33:08.975338 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975268    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:33:08.975548 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:08.975531    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078\": container with ID starting with ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078 not found: ID does not exist" containerID="ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"
Jul 02 13:33:08.975592 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975555    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078"} err="failed to get container status \"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078\": rpc error: code = NotFound desc = could not find container \"ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078\": container with ID starting with ad123a0fdf26287aa747b4aa16a851d0a548eaac0954358ed2410a16ed359078 not found: ID does not exist"
Jul 02 13:33:08.975592 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975571    2565 scope.go:117] "RemoveContainer" containerID="4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"
Jul 02 13:33:08.975789 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:08.975774    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534\": container with ID starting with 4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534 not found: ID does not exist" containerID="4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"
Jul 02 13:33:08.975833 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975793    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534"} err="failed to get container status \"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534\": rpc error: code = NotFound desc = could not find container \"4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534\": container with ID starting with 4015af3c79e8cb16aee0fe8f5e8123176cd9a7ce30b83acce0e054618bd5b534 not found: ID does not exist"
Jul 02 13:33:08.975833 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.975805    2565 scope.go:117] "RemoveContainer" containerID="464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"
Jul 02 13:33:08.976055 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:08.976039    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2\": container with ID starting with 464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2 not found: ID does not exist" containerID="464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"
Jul 02 13:33:08.976116 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.976057    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2"} err="failed to get container status \"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2\": rpc error: code = NotFound desc = could not find container \"464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2\": container with ID starting with 464132af3bfc9bf4d3a2d26fb4d59e86c97a5212da231945d19151ef1f50d6e2 not found: ID does not exist"
Jul 02 13:33:08.980220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:08.980199    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2456a-predictor-798988c569-ksgsh"]
Jul 02 13:33:09.009783 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.009762    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2vf\" (UniqueName: \"kubernetes.io/projected/22b44a3a-d286-4537-b959-b90727a4eace-kube-api-access-ph2vf\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:09.009783 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.009783    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b44a3a-d286-4537-b959-b90727a4eace-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:09.009904 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.009793    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b44a3a-d286-4537-b959-b90727a4eace-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:09.815388 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.815367    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:33:09.918186 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config\") pod \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") "
Jul 02 13:33:09.918308 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") pod \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") "
Jul 02 13:33:09.918308 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert\") pod \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") "
Jul 02 13:33:09.918308 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918254    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vcd9\" (UniqueName: \"kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9\") pod \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") "
Jul 02 13:33:09.918308 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location\") pod \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\" (UID: \"eb27b97f-5640-4afa-bd00-de5fdd3ea1db\") "
Jul 02 13:33:09.918644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918614    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb27b97f-5640-4afa-bd00-de5fdd3ea1db" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:33:09.918751 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918614    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb27b97f-5640-4afa-bd00-de5fdd3ea1db" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:33:09.918751 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.918658    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb27b97f-5640-4afa-bd00-de5fdd3ea1db" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:33:09.920349 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.920323    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9" (OuterVolumeSpecName: "kube-api-access-6vcd9") pod "eb27b97f-5640-4afa-bd00-de5fdd3ea1db" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db"). InnerVolumeSpecName "kube-api-access-6vcd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:33:09.920349 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.920333    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb27b97f-5640-4afa-bd00-de5fdd3ea1db" (UID: "eb27b97f-5640-4afa-bd00-de5fdd3ea1db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:33:09.957365 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.957341    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerID="8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78" exitCode=0
Jul 02 13:33:09.957668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.957392    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"
Jul 02 13:33:09.957668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.957425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerDied","Data":"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"}
Jul 02 13:33:09.957668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.957477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv" event={"ID":"eb27b97f-5640-4afa-bd00-de5fdd3ea1db","Type":"ContainerDied","Data":"52a83d703979e2a21c5d3bd9d7293a5ce168bbe8f659e328d01e7f001f32e8ee"}
Jul 02 13:33:09.957668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.957503    2565 scope.go:117] "RemoveContainer" containerID="dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"
Jul 02 13:33:09.971545 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.971526    2565 scope.go:117] "RemoveContainer" containerID="8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"
Jul 02 13:33:09.978887 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.978869    2565 scope.go:117] "RemoveContainer" containerID="5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"
Jul 02 13:33:09.984574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.984554    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:33:09.986689 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.986673    2565 scope.go:117] "RemoveContainer" containerID="dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"
Jul 02 13:33:09.987495 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:09.987455    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38\": container with ID starting with dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38 not found: ID does not exist" containerID="dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"
Jul 02 13:33:09.987611 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.987504    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38"} err="failed to get container status \"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38\": rpc error: code = NotFound desc = could not find container \"dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38\": container with ID starting with dade13cd2d28068630671817c6542c4e882eda26ccaecdf93cdaa045f4f58b38 not found: ID does not exist"
Jul 02 13:33:09.987611 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.987531    2565 scope.go:117] "RemoveContainer" containerID="8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"
Jul 02 13:33:09.987823 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:09.987806    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78\": container with ID starting with 8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78 not found: ID does not exist" containerID="8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"
Jul 02 13:33:09.987873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.987829    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78"} err="failed to get container status \"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78\": rpc error: code = NotFound desc = could not find container \"8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78\": container with ID starting with 8318ce82d2798a0f6b89ea25951a38352cd870c3f4a5af10efced02d68134f78 not found: ID does not exist"
Jul 02 13:33:09.987873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.987844    2565 scope.go:117] "RemoveContainer" containerID="5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"
Jul 02 13:33:09.988114 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:33:09.988097    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d\": container with ID starting with 5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d not found: ID does not exist" containerID="5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"
Jul 02 13:33:09.988170 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.988118    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d"} err="failed to get container status \"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d\": rpc error: code = NotFound desc = could not find container \"5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d\": container with ID starting with 5e460bb97eab445bd7715f19463637d6f7792b8dc76a7b40e5d43eb72b37e24d not found: ID does not exist"
Jul 02 13:33:09.988731 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:09.988713    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2456a-predictor-7d5dfdb5b6-tmpgv"]
Jul 02 13:33:10.019618 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.019595    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.019618 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.019616    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.019712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.019628    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.019712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.019638    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.019712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.019646    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcd9\" (UniqueName: \"kubernetes.io/projected/eb27b97f-5640-4afa-bd00-de5fdd3ea1db-kube-api-access-6vcd9\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:33:10.707331 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.707297    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b44a3a-d286-4537-b959-b90727a4eace" path="/var/lib/kubelet/pods/22b44a3a-d286-4537-b959-b90727a4eace/volumes"
Jul 02 13:33:10.707807 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:10.707794    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" path="/var/lib/kubelet/pods/eb27b97f-5640-4afa-bd00-de5fdd3ea1db/volumes"
Jul 02 13:33:14.961718 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:14.961687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:33:25.155189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.155151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:25.206294 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206266    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:33:25.206595 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206582    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container"
Jul 02 13:33:25.206672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206598    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container"
Jul 02 13:33:25.206672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206608    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206613    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206625    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="storage-initializer"
Jul 02 13:33:25.206672 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206634    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="storage-initializer"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206678    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="storage-initializer"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206687    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="storage-initializer"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206696    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206737    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206746    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206843    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kserve-container"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206858    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206868    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb27b97f-5640-4afa-bd00-de5fdd3ea1db" containerName="kube-rbac-proxy"
Jul 02 13:33:25.206921 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.206877    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b44a3a-d286-4537-b959-b90727a4eace" containerName="kserve-container"
Jul 02 13:33:25.210254 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.210235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.212421 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.212404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:33:25.212733 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.212713    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:33:25.212846 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.212803    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-260d5-kube-rbac-proxy-sar-config\""
Jul 02 13:33:25.212846 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.212816    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-260d5-predictor-serving-cert\""
Jul 02 13:33:25.222065 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.222040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:33:25.226625 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.226598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.226919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.226885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5vb\" (UniqueName: \"kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.227055 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.227006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.227055 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.227046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.227196 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.227175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328065 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5vb\" (UniqueName: \"kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328546 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328524    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328738 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328713    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.328823 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.328787    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.330562 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.330543    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.336922 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.336904    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5vb\" (UniqueName: \"kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb\") pod \"isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.363033 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.363007    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:25.526687 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.526664    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:25.654124 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:25.654101    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:33:25.656062 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:33:25.656035    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f0d9a8_2516_4512_be58_8f775a4afcf6.slice/crio-604f9c6ea4bc9f8cae3a377beb8c75f09fde2b4edfb444879eae99fb13550632 WatchSource:0}: Error finding container 604f9c6ea4bc9f8cae3a377beb8c75f09fde2b4edfb444879eae99fb13550632: Status 404 returned error can't find the container with id 604f9c6ea4bc9f8cae3a377beb8c75f09fde2b4edfb444879eae99fb13550632
Jul 02 13:33:26.021746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:26.021667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerStarted","Data":"39b4b8a619cc0710b2b2efa17ed2867e2acebcad165622f07a407643483ca6e9"}
Jul 02 13:33:26.021746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:26.021730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerStarted","Data":"604f9c6ea4bc9f8cae3a377beb8c75f09fde2b4edfb444879eae99fb13550632"}
Jul 02 13:33:30.041462 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:30.041430    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerID="39b4b8a619cc0710b2b2efa17ed2867e2acebcad165622f07a407643483ca6e9" exitCode=0
Jul 02 13:33:30.041838 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:30.041502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerDied","Data":"39b4b8a619cc0710b2b2efa17ed2867e2acebcad165622f07a407643483ca6e9"}
Jul 02 13:33:31.049990 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.049954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerStarted","Data":"91c9dc1cab2eba1cdb018d00fc31d0ecc837d0f7bb7b3c20dac8af30ad1d416f"}
Jul 02 13:33:31.050351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.050024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerStarted","Data":"0bf79b6ecec65eadd6b5ef1ff6f97d8c5c0e5bdea1ce6787393fe47dd008fc2b"}
Jul 02 13:33:31.050351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.050049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerStarted","Data":"a7e20db00111177d95c0ec1cf78cc4fd3b9f8fe2c1f2a471b18777a4469114cd"}
Jul 02 13:33:31.050351 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.050243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:31.072632 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.072591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podStartSLOduration=2.409481507 podStartE2EDuration="6.072578344s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663096837s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:33:31.070733009 +0000 UTC m=+1116.740540387" watchObservedRunningTime="2026-07-02 13:33:31.072578344 +0000 UTC m=+1116.742385711"
Jul 02 13:33:31.835123 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:31.835088    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:32.056150 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:32.056110    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:32.056511 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:32.056186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:32.057553 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:32.057517    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:33:32.058215 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:32.058181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:33:33.061535 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:33.061493    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:33:33.062063 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:33.062038    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:33:38.065416 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:38.065388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:33:38.065970 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:38.065922    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:33:38.066275 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:38.066256    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:33:48.066427 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:48.066381    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:33:48.066866 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:48.066843    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:33:58.066739 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:58.066699    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:33:58.067242 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:33:58.067164    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:08.066398 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:08.066355    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:34:08.066852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:08.066825    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:18.065954 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:18.065893    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:34:18.066397 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:18.066316    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:28.066382 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:28.066342    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:34:28.066861 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:28.066836    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:38.066341 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:38.066299    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:34:38.066774 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:38.066670    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:44.708490 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:44.708452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:34:44.708925 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:44.708548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:34:50.176767 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.176738    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx_a986669e-f600-413f-bcf4-132fbcb39dd2/kserve-container/0.log"
Jul 02 13:34:50.374924 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.374894    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:34:50.375500 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.375471    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" containerID="cri-o://a7e20db00111177d95c0ec1cf78cc4fd3b9f8fe2c1f2a471b18777a4469114cd" gracePeriod=30
Jul 02 13:34:50.375759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.375569    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" containerID="cri-o://0bf79b6ecec65eadd6b5ef1ff6f97d8c5c0e5bdea1ce6787393fe47dd008fc2b" gracePeriod=30
Jul 02 13:34:50.376918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.376172    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" containerID="cri-o://91c9dc1cab2eba1cdb018d00fc31d0ecc837d0f7bb7b3c20dac8af30ad1d416f" gracePeriod=30
Jul 02 13:34:50.389683 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.386790    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:50.431271 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.431190    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:34:50.434896 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.434874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.437287 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.437264    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dc1a2-predictor-serving-cert\""
Jul 02 13:34:50.437407 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.437389    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dc1a2-kube-rbac-proxy-sar-config\""
Jul 02 13:34:50.449235 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.449212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:34:50.476050 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.476017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:50.490800 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.490767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.491000 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.490812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.491000 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.490850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.491000 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.490916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.491211 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.491020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8ch\" (UniqueName: \"kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.492732 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.492697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:34:50.493000 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.492970    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kserve-container" containerID="cri-o://a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b" gracePeriod=30
Jul 02 13:34:50.493162 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.493008    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kube-rbac-proxy" containerID="cri-o://37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9" gracePeriod=30
Jul 02 13:34:50.547855 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.547822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:50.591797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.591765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.591816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8ch\" (UniqueName: \"kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.591858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.591881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592013 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.591903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592350 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.592331    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592745 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.592722    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.592783 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.592722    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.594270 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.594248    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.600883 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.600858    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8ch\" (UniqueName: \"kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch\") pod \"isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.729387 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.729364    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:34:50.745479 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.745426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:50.793570 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.793524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhwql\" (UniqueName: \"kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql\") pod \"a986669e-f600-413f-bcf4-132fbcb39dd2\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") "
Jul 02 13:34:50.793713 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.793648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config\") pod \"a986669e-f600-413f-bcf4-132fbcb39dd2\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") "
Jul 02 13:34:50.793713 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.793692    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls\") pod \"a986669e-f600-413f-bcf4-132fbcb39dd2\" (UID: \"a986669e-f600-413f-bcf4-132fbcb39dd2\") "
Jul 02 13:34:50.794693 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.794643    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a986669e-f600-413f-bcf4-132fbcb39dd2" (UID: "a986669e-f600-413f-bcf4-132fbcb39dd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:34:50.796498 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.796464    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a986669e-f600-413f-bcf4-132fbcb39dd2" (UID: "a986669e-f600-413f-bcf4-132fbcb39dd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:34:50.796617 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.796520    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql" (OuterVolumeSpecName: "kube-api-access-nhwql") pod "a986669e-f600-413f-bcf4-132fbcb39dd2" (UID: "a986669e-f600-413f-bcf4-132fbcb39dd2"). InnerVolumeSpecName "kube-api-access-nhwql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:34:50.870652 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.870628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:34:50.871965 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:34:50.871927    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca25eb9_d41f_4345_9118_03d215ad2307.slice/crio-2af7da5b903b43b0d4597775e6b5a9687d528782333b2848b71c961b22b17a59 WatchSource:0}: Error finding container 2af7da5b903b43b0d4597775e6b5a9687d528782333b2848b71c961b22b17a59: Status 404 returned error can't find the container with id 2af7da5b903b43b0d4597775e6b5a9687d528782333b2848b71c961b22b17a59
Jul 02 13:34:50.894740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.894718    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:34:50.894822 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.894751    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a986669e-f600-413f-bcf4-132fbcb39dd2-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:34:50.894822 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:50.894761    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwql\" (UniqueName: \"kubernetes.io/projected/a986669e-f600-413f-bcf4-132fbcb39dd2-kube-api-access-nhwql\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:34:51.386436 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.386402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerStarted","Data":"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"}
Jul 02 13:34:51.386878 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.386469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerStarted","Data":"2af7da5b903b43b0d4597775e6b5a9687d528782333b2848b71c961b22b17a59"}
Jul 02 13:34:51.388055 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388025    2565 generic.go:356] "Generic (PLEG): container finished" podID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerID="37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9" exitCode=2
Jul 02 13:34:51.388055 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388052    2565 generic.go:356] "Generic (PLEG): container finished" podID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerID="a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b" exitCode=2
Jul 02 13:34:51.388230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"
Jul 02 13:34:51.388230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerDied","Data":"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"}
Jul 02 13:34:51.388230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerDied","Data":"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"}
Jul 02 13:34:51.388230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx" event={"ID":"a986669e-f600-413f-bcf4-132fbcb39dd2","Type":"ContainerDied","Data":"cd8382693732b344954fbb6c7b54166a346d31082725a3819ec80b06508ac284"}
Jul 02 13:34:51.388230 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.388214    2565 scope.go:117] "RemoveContainer" containerID="37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"
Jul 02 13:34:51.390913 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.390889    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerID="0bf79b6ecec65eadd6b5ef1ff6f97d8c5c0e5bdea1ce6787393fe47dd008fc2b" exitCode=2
Jul 02 13:34:51.391073 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.390925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerDied","Data":"0bf79b6ecec65eadd6b5ef1ff6f97d8c5c0e5bdea1ce6787393fe47dd008fc2b"}
Jul 02 13:34:51.406614 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.406596    2565 scope.go:117] "RemoveContainer" containerID="a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"
Jul 02 13:34:51.414882 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.414860    2565 scope.go:117] "RemoveContainer" containerID="37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"
Jul 02 13:34:51.415183 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:34:51.415160    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9\": container with ID starting with 37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9 not found: ID does not exist" containerID="37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"
Jul 02 13:34:51.415246 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415193    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"} err="failed to get container status \"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9\": rpc error: code = NotFound desc = could not find container \"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9\": container with ID starting with 37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9 not found: ID does not exist"
Jul 02 13:34:51.415246 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415211    2565 scope.go:117] "RemoveContainer" containerID="a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"
Jul 02 13:34:51.415476 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:34:51.415456    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b\": container with ID starting with a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b not found: ID does not exist" containerID="a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"
Jul 02 13:34:51.415532 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415485    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"} err="failed to get container status \"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b\": rpc error: code = NotFound desc = could not find container \"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b\": container with ID starting with a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b not found: ID does not exist"
Jul 02 13:34:51.415532 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415502    2565 scope.go:117] "RemoveContainer" containerID="37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"
Jul 02 13:34:51.415712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415695    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9"} err="failed to get container status \"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9\": rpc error: code = NotFound desc = could not find container \"37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9\": container with ID starting with 37b1a48d37d28651ea61bab763421a82ffbf58cf32ff7fa4dce04e9eb97fb1c9 not found: ID does not exist"
Jul 02 13:34:51.415712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415711    2565 scope.go:117] "RemoveContainer" containerID="a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"
Jul 02 13:34:51.415962 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.415927    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b"} err="failed to get container status \"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b\": rpc error: code = NotFound desc = could not find container \"a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b\": container with ID starting with a631abd2c97006bebdfce43c0644cdd0f97edff30abfadb753052b23dc05e39b not found: ID does not exist"
Jul 02 13:34:51.421081 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.421040    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:34:51.422829 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:51.422810    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-260d5-predictor-5cfcbb6d5f-g2ntx"]
Jul 02 13:34:52.707045 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:52.707013    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" path="/var/lib/kubelet/pods/a986669e-f600-413f-bcf4-132fbcb39dd2/volumes"
Jul 02 13:34:53.062689 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:53.062648    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:34:54.705997 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:54.705962    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:34:54.706403 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:54.706338    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:34:54.730655 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:54.730627    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:34:54.733957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:54.733924    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:34:55.412739 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:55.412706    2565 generic.go:356] "Generic (PLEG): container finished" podID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerID="1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50" exitCode=0
Jul 02 13:34:55.412914 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:55.412781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerDied","Data":"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"}
Jul 02 13:34:55.415235 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:55.415214    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerID="a7e20db00111177d95c0ec1cf78cc4fd3b9f8fe2c1f2a471b18777a4469114cd" exitCode=0
Jul 02 13:34:55.415349 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:55.415239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerDied","Data":"a7e20db00111177d95c0ec1cf78cc4fd3b9f8fe2c1f2a471b18777a4469114cd"}
Jul 02 13:34:56.420993 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:56.420961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerStarted","Data":"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"}
Jul 02 13:34:56.421364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:56.421020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerStarted","Data":"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"}
Jul 02 13:34:56.421364 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:56.421206    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:56.440212 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:56.440163    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podStartSLOduration=2.69912226 podStartE2EDuration="6.440148844s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741026584s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:34:56.438231456 +0000 UTC m=+1202.108038842" watchObservedRunningTime="2026-07-02 13:34:56.440148844 +0000 UTC m=+1202.109956211"
Jul 02 13:34:57.425674 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:57.425633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:34:57.426879 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:57.426852    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:34:58.062692 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:58.062649    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:34:58.429396 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:34:58.429295    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:03.062288 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:03.062247    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:35:03.062723 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:03.062369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:35:03.433692 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:03.433617    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:35:03.434078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:03.434052    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:04.705819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:04.705771    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:35:04.706248 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:04.706145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:35:08.062746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:08.062704    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:35:13.062494 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:13.062447    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:35:13.434848 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:13.434761    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:14.705832 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:14.705792    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:35:14.706287 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:14.706154    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:35:14.706505 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:14.706489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:35:14.706589 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:14.706580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:35:18.062342 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:18.062304    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:35:20.521477 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.521441    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerID="91c9dc1cab2eba1cdb018d00fc31d0ecc837d0f7bb7b3c20dac8af30ad1d416f" exitCode=137
Jul 02 13:35:20.521826 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.521482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerDied","Data":"91c9dc1cab2eba1cdb018d00fc31d0ecc837d0f7bb7b3c20dac8af30ad1d416f"}
Jul 02 13:35:20.560272 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.560246    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:35:20.631708 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.631684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls\") pod \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") "
Jul 02 13:35:20.631856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.631727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config\") pod \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") "
Jul 02 13:35:20.631856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.631808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location\") pod \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") "
Jul 02 13:35:20.631856 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.631837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g5vb\" (UniqueName: \"kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb\") pod \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") "
Jul 02 13:35:20.632047 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.631907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert\") pod \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\" (UID: \"b4f0d9a8-2516-4512-be58-8f775a4afcf6\") "
Jul 02 13:35:20.632217 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.632128    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4f0d9a8-2516-4512-be58-8f775a4afcf6" (UID: "b4f0d9a8-2516-4512-be58-8f775a4afcf6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:35:20.632285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.632194    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4f0d9a8-2516-4512-be58-8f775a4afcf6" (UID: "b4f0d9a8-2516-4512-be58-8f775a4afcf6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:35:20.632285 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.632266    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4f0d9a8-2516-4512-be58-8f775a4afcf6" (UID: "b4f0d9a8-2516-4512-be58-8f775a4afcf6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:35:20.633830 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.633787    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4f0d9a8-2516-4512-be58-8f775a4afcf6" (UID: "b4f0d9a8-2516-4512-be58-8f775a4afcf6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:35:20.633830 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.633803    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb" (OuterVolumeSpecName: "kube-api-access-5g5vb") pod "b4f0d9a8-2516-4512-be58-8f775a4afcf6" (UID: "b4f0d9a8-2516-4512-be58-8f775a4afcf6"). InnerVolumeSpecName "kube-api-access-5g5vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:35:20.732909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.732887    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:35:20.732909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.732908    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0d9a8-2516-4512-be58-8f775a4afcf6-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:35:20.733056 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.732917    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:35:20.733056 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.732928    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:35:20.733056 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:20.732961    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5vb\" (UniqueName: \"kubernetes.io/projected/b4f0d9a8-2516-4512-be58-8f775a4afcf6-kube-api-access-5g5vb\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:35:21.528320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.528292    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"
Jul 02 13:35:21.528320 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.528306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26" event={"ID":"b4f0d9a8-2516-4512-be58-8f775a4afcf6","Type":"ContainerDied","Data":"604f9c6ea4bc9f8cae3a377beb8c75f09fde2b4edfb444879eae99fb13550632"}
Jul 02 13:35:21.528826 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.528400    2565 scope.go:117] "RemoveContainer" containerID="91c9dc1cab2eba1cdb018d00fc31d0ecc837d0f7bb7b3c20dac8af30ad1d416f"
Jul 02 13:35:21.538822 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.538804    2565 scope.go:117] "RemoveContainer" containerID="0bf79b6ecec65eadd6b5ef1ff6f97d8c5c0e5bdea1ce6787393fe47dd008fc2b"
Jul 02 13:35:21.546189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.546172    2565 scope.go:117] "RemoveContainer" containerID="a7e20db00111177d95c0ec1cf78cc4fd3b9f8fe2c1f2a471b18777a4469114cd"
Jul 02 13:35:21.550686 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.550666    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:35:21.553994 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.553974    2565 scope.go:117] "RemoveContainer" containerID="39b4b8a619cc0710b2b2efa17ed2867e2acebcad165622f07a407643483ca6e9"
Jul 02 13:35:21.555104 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:21.555087    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-260d5-predictor-6f98b5899b-5pp26"]
Jul 02 13:35:22.706680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:22.706648    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" path="/var/lib/kubelet/pods/b4f0d9a8-2516-4512-be58-8f775a4afcf6/volumes"
Jul 02 13:35:23.434369 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:23.434332    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:33.434998 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:33.434957    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:43.434376 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:43.434337    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:35:53.434579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:35:53.434493    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:03.434471 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:03.434429    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:07.047374 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:07.047333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:36:13.434536 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:13.434499    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:23.434831 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:23.434787    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:30.707220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:30.707177    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:40.707189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:40.707148    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:36:50.708125 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:36:50.708086    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:37:00.707771 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:00.707727    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:37:10.707127 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:10.707090    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:37:20.707321 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:20.707280    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:37:30.708086 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:30.708051    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:37:40.649152 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:40.649114    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:37:40.649712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:40.649445    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" containerID="cri-o://b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7" gracePeriod=30
Jul 02 13:37:40.649712 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:40.649513    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy" containerID="cri-o://236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04" gracePeriod=30
Jul 02 13:37:40.689189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:40.689154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:40.707493 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:40.707463    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 13:37:41.067792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:41.067757    2565 generic.go:356] "Generic (PLEG): container finished" podID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerID="236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04" exitCode=2
Jul 02 13:37:41.067976 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:41.067815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerDied","Data":"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"}
Jul 02 13:37:43.429907 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:43.429868    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 13:37:45.713280 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.713245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:45.773510 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773475    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:37:45.773799 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kube-rbac-proxy"
Jul 02 13:37:45.773849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773800    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kube-rbac-proxy"
Jul 02 13:37:45.773849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773811    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container"
Jul 02 13:37:45.773849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773816    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container"
Jul 02 13:37:45.773849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kserve-container"
Jul 02 13:37:45.773849 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773852    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kserve-container"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773861    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773867    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773874    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773879    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773887    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="storage-initializer"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773892    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="storage-initializer"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773963    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kube-rbac-proxy"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773974    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kserve-container"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773980    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="agent"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773986    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a986669e-f600-413f-bcf4-132fbcb39dd2" containerName="kube-rbac-proxy"
Jul 02 13:37:45.774051 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.773991    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0d9a8-2516-4512-be58-8f775a4afcf6" containerName="kserve-container"
Jul 02 13:37:45.777050 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.777030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.779511 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.779487    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b32985-predictor-serving-cert\""
Jul 02 13:37:45.779511 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.779507    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b32985-kube-rbac-proxy-sar-config\""
Jul 02 13:37:45.786574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.786549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:37:45.849080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.849046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:45.875698 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.875669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvsm\" (UniqueName: \"kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.875836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.875704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.875836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.875732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.875836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.875773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.875836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.875829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977255 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977255 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977473 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977473 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvsm\" (UniqueName: \"kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977473 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977622 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977590    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977978 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977955    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.977978 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.977968    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.979659 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.979637    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:45.986600 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:45.986580    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvsm\" (UniqueName: \"kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm\") pod \"isvc-primary-b32985-predictor-7885987955-55478\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") " pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:46.088311 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:46.088280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:46.215746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:46.215714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:37:46.218998 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:37:46.218970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc769812_88ad_4d6f_8b33_17f8cae4a82b.slice/crio-6a6720f12703fcc495fe64dddcef965f8838afe37c60bc75cd493701dbbc4d7a WatchSource:0}: Error finding container 6a6720f12703fcc495fe64dddcef965f8838afe37c60bc75cd493701dbbc4d7a: Status 404 returned error can't find the container with id 6a6720f12703fcc495fe64dddcef965f8838afe37c60bc75cd493701dbbc4d7a
Jul 02 13:37:47.093625 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:47.093586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerStarted","Data":"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"}
Jul 02 13:37:47.094022 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:47.093655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerStarted","Data":"6a6720f12703fcc495fe64dddcef965f8838afe37c60bc75cd493701dbbc4d7a"}
Jul 02 13:37:48.430405 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:48.430364    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 13:37:50.106121 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.106091    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerID="aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab" exitCode=0
Jul 02 13:37:50.106440 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.106163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerDied","Data":"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"}
Jul 02 13:37:50.296185 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.296166    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:37:50.413537 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location\") pod \"8ca25eb9-d41f-4345-9118-03d215ad2307\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") "
Jul 02 13:37:50.413734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413577    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config\") pod \"8ca25eb9-d41f-4345-9118-03d215ad2307\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") "
Jul 02 13:37:50.413734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls\") pod \"8ca25eb9-d41f-4345-9118-03d215ad2307\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") "
Jul 02 13:37:50.413734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413656    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert\") pod \"8ca25eb9-d41f-4345-9118-03d215ad2307\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") "
Jul 02 13:37:50.413734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413694    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn8ch\" (UniqueName: \"kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch\") pod \"8ca25eb9-d41f-4345-9118-03d215ad2307\" (UID: \"8ca25eb9-d41f-4345-9118-03d215ad2307\") "
Jul 02 13:37:50.413980 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413859    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ca25eb9-d41f-4345-9118-03d215ad2307" (UID: "8ca25eb9-d41f-4345-9118-03d215ad2307"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:50.414038 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.413986    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ca25eb9-d41f-4345-9118-03d215ad2307" (UID: "8ca25eb9-d41f-4345-9118-03d215ad2307"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:37:50.414081 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.414033    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ca25eb9-d41f-4345-9118-03d215ad2307" (UID: "8ca25eb9-d41f-4345-9118-03d215ad2307"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:37:50.415748 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.415722    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch" (OuterVolumeSpecName: "kube-api-access-pn8ch") pod "8ca25eb9-d41f-4345-9118-03d215ad2307" (UID: "8ca25eb9-d41f-4345-9118-03d215ad2307"). InnerVolumeSpecName "kube-api-access-pn8ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:37:50.415748 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.415724    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ca25eb9-d41f-4345-9118-03d215ad2307" (UID: "8ca25eb9-d41f-4345-9118-03d215ad2307"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:37:50.514619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.514515    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ca25eb9-d41f-4345-9118-03d215ad2307-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:37:50.514619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.514561    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:37:50.514619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.514572    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8ch\" (UniqueName: \"kubernetes.io/projected/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-api-access-pn8ch\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:37:50.514619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.514581    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca25eb9-d41f-4345-9118-03d215ad2307-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:37:50.514619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:50.514590    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ca25eb9-d41f-4345-9118-03d215ad2307-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:37:51.111515 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.111481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerStarted","Data":"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"}
Jul 02 13:37:51.111922 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.111551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerStarted","Data":"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"}
Jul 02 13:37:51.111989 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.111921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:51.113253 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.113225    2565 generic.go:356] "Generic (PLEG): container finished" podID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerID="b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7" exitCode=0
Jul 02 13:37:51.113367 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.113279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"
Jul 02 13:37:51.113367 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.113291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerDied","Data":"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"}
Jul 02 13:37:51.113367 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.113345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2" event={"ID":"8ca25eb9-d41f-4345-9118-03d215ad2307","Type":"ContainerDied","Data":"2af7da5b903b43b0d4597775e6b5a9687d528782333b2848b71c961b22b17a59"}
Jul 02 13:37:51.113467 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.113383    2565 scope.go:117] "RemoveContainer" containerID="236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"
Jul 02 13:37:51.121769 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.121750    2565 scope.go:117] "RemoveContainer" containerID="b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"
Jul 02 13:37:51.128613 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.128597    2565 scope.go:117] "RemoveContainer" containerID="1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"
Jul 02 13:37:51.132681 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.132640    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podStartSLOduration=2.462500952 podStartE2EDuration="6.132627879s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670126927s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:37:51.130895289 +0000 UTC m=+1376.800702660" watchObservedRunningTime="2026-07-02 13:37:51.132627879 +0000 UTC m=+1376.802435247"
Jul 02 13:37:51.136197 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.136183    2565 scope.go:117] "RemoveContainer" containerID="236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"
Jul 02 13:37:51.136475 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:37:51.136450    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04\": container with ID starting with 236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04 not found: ID does not exist" containerID="236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"
Jul 02 13:37:51.136542 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.136484    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04"} err="failed to get container status \"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04\": rpc error: code = NotFound desc = could not find container \"236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04\": container with ID starting with 236778babe5cc1b77dae164ebbc8bfab5b627c13ac389c4252591fbe2743aa04 not found: ID does not exist"
Jul 02 13:37:51.136542 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.136505    2565 scope.go:117] "RemoveContainer" containerID="b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"
Jul 02 13:37:51.136814 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:37:51.136794    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7\": container with ID starting with b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7 not found: ID does not exist" containerID="b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"
Jul 02 13:37:51.136909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.136821    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7"} err="failed to get container status \"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7\": rpc error: code = NotFound desc = could not find container \"b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7\": container with ID starting with b65239fc3ca936f3e95b27616319ace35e3b62c57affb45641bc2aa1042887b7 not found: ID does not exist"
Jul 02 13:37:51.136909 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.136836    2565 scope.go:117] "RemoveContainer" containerID="1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"
Jul 02 13:37:51.137122 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:37:51.137101    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50\": container with ID starting with 1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50 not found: ID does not exist" containerID="1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"
Jul 02 13:37:51.137165 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.137129    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50"} err="failed to get container status \"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50\": rpc error: code = NotFound desc = could not find container \"1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50\": container with ID starting with 1d000a0ab43aaa77d7dfaf4a7cbfd7ed1f90f4040f8aea353221d1609ef29d50 not found: ID does not exist"
Jul 02 13:37:51.145283 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.145258    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:37:51.149391 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:51.149371    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dc1a2-predictor-65df7698b4-grgh2"]
Jul 02 13:37:52.118416 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:52.118385    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:52.119629 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:52.119600    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:37:52.707701 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:52.707671    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" path="/var/lib/kubelet/pods/8ca25eb9-d41f-4345-9118-03d215ad2307/volumes"
Jul 02 13:37:53.122085 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:53.122046    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:37:58.126337 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:58.126304    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:37:58.126796 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:37:58.126770    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:07.860556 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:07.860519    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:38:08.127041 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:08.126927    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:18.126799 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:18.126759    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:28.126780 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:28.126738    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:38.127590 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:38.127546    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:48.126977 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:48.126876    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:38:58.127339 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:38:58.127301    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:39:00.702180 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:00.702137    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:39:10.707676 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:10.707647    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:39:15.932161 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.932082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:15.982564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982529    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:15.982842 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982828    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container"
Jul 02 13:39:15.982906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982843    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container"
Jul 02 13:39:15.982906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982851    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy"
Jul 02 13:39:15.982906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy"
Jul 02 13:39:15.982906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982880    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="storage-initializer"
Jul 02 13:39:15.982906 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982886    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="storage-initializer"
Jul 02 13:39:15.983252 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982954    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kube-rbac-proxy"
Jul 02 13:39:15.983252 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.982965    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca25eb9-d41f-4345-9118-03d215ad2307" containerName="kserve-container"
Jul 02 13:39:15.986260 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.986236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:15.988504 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.988483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b32985-predictor-serving-cert\""
Jul 02 13:39:15.988504 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.988496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b32985-dockercfg-xx96q\""
Jul 02 13:39:15.988668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.988504    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b32985-kube-rbac-proxy-sar-config\""
Jul 02 13:39:15.988668 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.988632    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b32985\""
Jul 02 13:39:15.996223 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:15.996199    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:16.094972 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.094945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.095128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.094995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.095128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.095020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.095128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.095065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.095128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.095086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7hw\" (UniqueName: \"kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.195828 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.195740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.195828 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.195791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.195828 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.195815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.196117 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.195846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.196117 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.195872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7hw\" (UniqueName: \"kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.196117 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:16.195903    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-b32985-predictor-serving-cert: secret "isvc-secondary-b32985-predictor-serving-cert" not found
Jul 02 13:39:16.196117 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:16.196008    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls podName:f67abeb2-fab9-49c4-ae41-21ba4333356e nodeName:}" failed. No retries permitted until 2026-07-02 13:39:16.695988617 +0000 UTC m=+1462.365795968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls") pod "isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e") : secret "isvc-secondary-b32985-predictor-serving-cert" not found
Jul 02 13:39:16.196326 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.196276    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.196489 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.196472    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.196525 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.196484    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.204959 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.204914    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7hw\" (UniqueName: \"kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.302245 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.302214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:16.699306 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.699267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.701598 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.701573    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") pod \"isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") " pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:16.897462 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:16.897428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:17.025308 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:17.025286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:17.026883 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:39:17.026856    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67abeb2_fab9_49c4_ae41_21ba4333356e.slice/crio-fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38 WatchSource:0}: Error finding container fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38: Status 404 returned error can't find the container with id fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38
Jul 02 13:39:17.461993 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:17.461961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerStarted","Data":"8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"}
Jul 02 13:39:17.462184 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:17.462017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerStarted","Data":"fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38"}
Jul 02 13:39:22.484094 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:22.484064    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/0.log"
Jul 02 13:39:22.484444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:22.484104    2565 generic.go:356] "Generic (PLEG): container finished" podID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerID="8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910" exitCode=1
Jul 02 13:39:22.484444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:22.484173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerDied","Data":"8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"}
Jul 02 13:39:23.490252 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:23.490222    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/0.log"
Jul 02 13:39:23.490618 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:23.490305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerStarted","Data":"40faf14090d6405117b3c50b96971969fc4d6923c00ca5304f1f0ccb521e2249"}
Jul 02 13:39:27.506369 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.506326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/1.log"
Jul 02 13:39:27.506702 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.506682    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/0.log"
Jul 02 13:39:27.506753 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.506711    2565 generic.go:356] "Generic (PLEG): container finished" podID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerID="40faf14090d6405117b3c50b96971969fc4d6923c00ca5304f1f0ccb521e2249" exitCode=1
Jul 02 13:39:27.506790 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.506778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerDied","Data":"40faf14090d6405117b3c50b96971969fc4d6923c00ca5304f1f0ccb521e2249"}
Jul 02 13:39:27.506845 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.506832    2565 scope.go:117] "RemoveContainer" containerID="8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"
Jul 02 13:39:27.507197 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.507172    2565 scope.go:117] "RemoveContainer" containerID="8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"
Jul 02 13:39:27.517107 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:27.517088    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_kserve-ci-e2e-test(f67abeb2-fab9-49c4-ae41-21ba4333356e)\"" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e"
Jul 02 13:39:27.517241 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:27.517209    2565 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_kserve-ci-e2e-test_f67abeb2-fab9-49c4-ae41-21ba4333356e_0 in pod sandbox fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38 from index: no such id: '8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910'" containerID="8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"
Jul 02 13:39:27.517293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:27.517251    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_kserve-ci-e2e-test_f67abeb2-fab9-49c4-ae41-21ba4333356e_0 in pod sandbox fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38 from index: no such id: '8caacc21a53df7bef36883185e54af04115fb9b04cfb4be3efc5fff0d997f910'"
Jul 02 13:39:28.512537 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:28.512507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/1.log"
Jul 02 13:39:34.063545 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.063508    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:34.084643 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.084608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:34.135792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.129157    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:39:34.135792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.129550    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container" containerID="cri-o://6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe" gracePeriod=30
Jul 02 13:39:34.135792 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.129827    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kube-rbac-proxy" containerID="cri-o://61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f" gracePeriod=30
Jul 02 13:39:34.151719 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.151659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:34.255834 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.255814    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/1.log"
Jul 02 13:39:34.257918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.257904    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:34.336755 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.336698    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config\") pod \"f67abeb2-fab9-49c4-ae41-21ba4333356e\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") "
Jul 02 13:39:34.336755 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.336734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location\") pod \"f67abeb2-fab9-49c4-ae41-21ba4333356e\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") "
Jul 02 13:39:34.336918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.336780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert\") pod \"f67abeb2-fab9-49c4-ae41-21ba4333356e\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") "
Jul 02 13:39:34.336918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.336801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") pod \"f67abeb2-fab9-49c4-ae41-21ba4333356e\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") "
Jul 02 13:39:34.336918 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.336835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv7hw\" (UniqueName: \"kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw\") pod \"f67abeb2-fab9-49c4-ae41-21ba4333356e\" (UID: \"f67abeb2-fab9-49c4-ae41-21ba4333356e\") "
Jul 02 13:39:34.337119 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.337040    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f67abeb2-fab9-49c4-ae41-21ba4333356e" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:34.337179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.337131    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67abeb2-fab9-49c4-ae41-21ba4333356e-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.337179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.337128    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f67abeb2-fab9-49c4-ae41-21ba4333356e" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:34.337179 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.337171    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f67abeb2-fab9-49c4-ae41-21ba4333356e" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:34.338793 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.338765    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw" (OuterVolumeSpecName: "kube-api-access-dv7hw") pod "f67abeb2-fab9-49c4-ae41-21ba4333356e" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e"). InnerVolumeSpecName "kube-api-access-dv7hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:34.338892 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.338865    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f67abeb2-fab9-49c4-ae41-21ba4333356e" (UID: "f67abeb2-fab9-49c4-ae41-21ba4333356e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:34.438362 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.438338    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.438362 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.438360    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67abeb2-fab9-49c4-ae41-21ba4333356e-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.438501 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.438368    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv7hw\" (UniqueName: \"kubernetes.io/projected/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-api-access-dv7hw\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.438501 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.438378    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67abeb2-fab9-49c4-ae41-21ba4333356e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.537220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.537190    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerID="61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f" exitCode=2
Jul 02 13:39:34.537328 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.537270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerDied","Data":"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"}
Jul 02 13:39:34.538464 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.538447    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk_f67abeb2-fab9-49c4-ae41-21ba4333356e/storage-initializer/1.log"
Jul 02 13:39:34.538542 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.538525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk" event={"ID":"f67abeb2-fab9-49c4-ae41-21ba4333356e","Type":"ContainerDied","Data":"fa65c00cc87beee69f530b80235923cfb6572fdb61e3b75eb61937eea9978e38"}
Jul 02 13:39:34.538581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.538530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"
Jul 02 13:39:34.538581 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.538570    2565 scope.go:117] "RemoveContainer" containerID="40faf14090d6405117b3c50b96971969fc4d6923c00ca5304f1f0ccb521e2249"
Jul 02 13:39:34.574371 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.574348    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:34.578617 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.578593    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b32985-predictor-8bdf6fb9b-j5rqk"]
Jul 02 13:39:34.707971 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:34.707884    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" path="/var/lib/kubelet/pods/f67abeb2-fab9-49c4-ae41-21ba4333356e/volumes"
Jul 02 13:39:38.122829 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.122792    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 13:39:38.481887 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.481864    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:39:38.557189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.557154    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerID="6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe" exitCode=0
Jul 02 13:39:38.557322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.557215    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"
Jul 02 13:39:38.557322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.557242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerDied","Data":"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"}
Jul 02 13:39:38.557322 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.557321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478" event={"ID":"fc769812-88ad-4d6f-8b33-17f8cae4a82b","Type":"ContainerDied","Data":"6a6720f12703fcc495fe64dddcef965f8838afe37c60bc75cd493701dbbc4d7a"}
Jul 02 13:39:38.557435 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.557348    2565 scope.go:117] "RemoveContainer" containerID="61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"
Jul 02 13:39:38.565082 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.565067    2565 scope.go:117] "RemoveContainer" containerID="6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"
Jul 02 13:39:38.570569 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570541    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config\") pod \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") "
Jul 02 13:39:38.570674 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert\") pod \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") "
Jul 02 13:39:38.570674 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location\") pod \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") "
Jul 02 13:39:38.570908 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls\") pod \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") "
Jul 02 13:39:38.570908 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wvsm\" (UniqueName: \"kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm\") pod \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\" (UID: \"fc769812-88ad-4d6f-8b33-17f8cae4a82b\") "
Jul 02 13:39:38.571039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570928    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc769812-88ad-4d6f-8b33-17f8cae4a82b" (UID: "fc769812-88ad-4d6f-8b33-17f8cae4a82b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:38.571039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.570980    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc769812-88ad-4d6f-8b33-17f8cae4a82b" (UID: "fc769812-88ad-4d6f-8b33-17f8cae4a82b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:38.571039 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.571008    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc769812-88ad-4d6f-8b33-17f8cae4a82b" (UID: "fc769812-88ad-4d6f-8b33-17f8cae4a82b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:38.571194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.571114    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:38.571194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.571128    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:38.571194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.571137    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:38.572243 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.572227    2565 scope.go:117] "RemoveContainer" containerID="aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"
Jul 02 13:39:38.572719 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.572697    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc769812-88ad-4d6f-8b33-17f8cae4a82b" (UID: "fc769812-88ad-4d6f-8b33-17f8cae4a82b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:38.572904 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.572883    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm" (OuterVolumeSpecName: "kube-api-access-9wvsm") pod "fc769812-88ad-4d6f-8b33-17f8cae4a82b" (UID: "fc769812-88ad-4d6f-8b33-17f8cae4a82b"). InnerVolumeSpecName "kube-api-access-9wvsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:38.585293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.585278    2565 scope.go:117] "RemoveContainer" containerID="61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"
Jul 02 13:39:38.585520 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:38.585502    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f\": container with ID starting with 61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f not found: ID does not exist" containerID="61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"
Jul 02 13:39:38.585567 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.585527    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f"} err="failed to get container status \"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f\": rpc error: code = NotFound desc = could not find container \"61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f\": container with ID starting with 61290bf7e468462b9ee2bac9ea1789413d110d34cf4a52fea51a503e35217b4f not found: ID does not exist"
Jul 02 13:39:38.585567 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.585543    2565 scope.go:117] "RemoveContainer" containerID="6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"
Jul 02 13:39:38.585758 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:38.585741    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe\": container with ID starting with 6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe not found: ID does not exist" containerID="6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"
Jul 02 13:39:38.585794 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.585764    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe"} err="failed to get container status \"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe\": rpc error: code = NotFound desc = could not find container \"6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe\": container with ID starting with 6ac256b0fcf6713d526f6c7aa27ee2dd46e766e083040ee96241d1ecc52724fe not found: ID does not exist"
Jul 02 13:39:38.585794 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.585779    2565 scope.go:117] "RemoveContainer" containerID="aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"
Jul 02 13:39:38.586016 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:38.585999    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab\": container with ID starting with aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab not found: ID does not exist" containerID="aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"
Jul 02 13:39:38.586073 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.586022    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab"} err="failed to get container status \"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab\": rpc error: code = NotFound desc = could not find container \"aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab\": container with ID starting with aa50fde021014f1dee4eb824e89ec165d1fb53d165df7da5b61a849328c436ab not found: ID does not exist"
Jul 02 13:39:38.672237 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.672180    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc769812-88ad-4d6f-8b33-17f8cae4a82b-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:38.672237 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.672201    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvsm\" (UniqueName: \"kubernetes.io/projected/fc769812-88ad-4d6f-8b33-17f8cae4a82b-kube-api-access-9wvsm\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:38.874818 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.874790    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:39:38.879161 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:38.879136    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b32985-predictor-7885987955-55478"]
Jul 02 13:39:39.201518 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.201482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:39.227875 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.227841    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:39.228206 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228189    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="storage-initializer"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228209    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="storage-initializer"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228230    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228239    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228255    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kube-rbac-proxy"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kube-rbac-proxy"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228276    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container"
Jul 02 13:39:39.228291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228284    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228365    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kube-rbac-proxy"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228376    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228387    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" containerName="kserve-container"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228469    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.228626 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.228573    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67abeb2-fab9-49c4-ae41-21ba4333356e" containerName="storage-initializer"
Jul 02 13:39:39.232952 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.232921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.235209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235188    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8b27ef-predictor-serving-cert\""
Jul 02 13:39:39.235422 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:39:39.235513 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235495    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:39:39.235576 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235495    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8b27ef-kube-rbac-proxy-sar-config\""
Jul 02 13:39:39.235576 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-8b27ef\""
Jul 02 13:39:39.235746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.235723    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:39:39.236306 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.236293    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-8b27ef-dockercfg-97vnb\""
Jul 02 13:39:39.240305 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.240284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:39.276895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.276872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.277078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.276900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjw2\" (UniqueName: \"kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.277078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.276918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.277194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.277096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.277194 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.277140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378207 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378331 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378331 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378331 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjw2\" (UniqueName: \"kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378491 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:39.378334    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-serving-cert: secret "isvc-init-fail-8b27ef-predictor-serving-cert" not found
Jul 02 13:39:39.378491 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378491 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:39.378405    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls podName:c7179d3b-bfae-4193-86c5-aafc42ae7b6b nodeName:}" failed. No retries permitted until 2026-07-02 13:39:39.878375639 +0000 UTC m=+1485.548182988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls") pod "isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b") : secret "isvc-init-fail-8b27ef-predictor-serving-cert" not found
Jul 02 13:39:39.378766 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378744    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.378898 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378881    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.379000 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.378978    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.387673 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.387651    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjw2\" (UniqueName: \"kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.624924 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.624899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:39.882862 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.882784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:39.885103 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:39.885075    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") pod \"isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:40.144915 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:40.144831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:40.272243 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:40.272153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:40.276770 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:39:40.276733    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7179d3b_bfae_4193_86c5_aafc42ae7b6b.slice/crio-3130523a5aa7379ac1390d9835fc5e66132535817daa1964c05ecd944bbadab5 WatchSource:0}: Error finding container 3130523a5aa7379ac1390d9835fc5e66132535817daa1964c05ecd944bbadab5: Status 404 returned error can't find the container with id 3130523a5aa7379ac1390d9835fc5e66132535817daa1964c05ecd944bbadab5
Jul 02 13:39:40.568023 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:40.567985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerStarted","Data":"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"}
Jul 02 13:39:40.568199 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:40.568049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerStarted","Data":"3130523a5aa7379ac1390d9835fc5e66132535817daa1964c05ecd944bbadab5"}
Jul 02 13:39:40.707895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:40.707858    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc769812-88ad-4d6f-8b33-17f8cae4a82b" path="/var/lib/kubelet/pods/fc769812-88ad-4d6f-8b33-17f8cae4a82b/volumes"
Jul 02 13:39:44.585664 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:44.585635    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/0.log"
Jul 02 13:39:44.586142 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:44.585676    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerID="63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528" exitCode=1
Jul 02 13:39:44.586142 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:44.585757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerDied","Data":"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"}
Jul 02 13:39:45.591700 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:45.591672    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/0.log"
Jul 02 13:39:45.592061 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:45.591717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerStarted","Data":"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"}
Jul 02 13:39:49.249645 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:49.249611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:49.250108 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:49.249902    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer" containerID="cri-o://a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55" gracePeriod=30
Jul 02 13:39:49.264374 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:49.264347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:50.597685 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.597662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/1.log"
Jul 02 13:39:50.598079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.598061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/0.log"
Jul 02 13:39:50.600376 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.600361    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:50.611919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.611901    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/1.log"
Jul 02 13:39:50.612291 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612277    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5_c7179d3b-bfae-4193-86c5-aafc42ae7b6b/storage-initializer/0.log"
Jul 02 13:39:50.612360 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612307    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerID="a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55" exitCode=1
Jul 02 13:39:50.612360 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerDied","Data":"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"}
Jul 02 13:39:50.612360 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612354    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"
Jul 02 13:39:50.612474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5" event={"ID":"c7179d3b-bfae-4193-86c5-aafc42ae7b6b","Type":"ContainerDied","Data":"3130523a5aa7379ac1390d9835fc5e66132535817daa1964c05ecd944bbadab5"}
Jul 02 13:39:50.612474 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.612403    2565 scope.go:117] "RemoveContainer" containerID="a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"
Jul 02 13:39:50.620128 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.620078    2565 scope.go:117] "RemoveContainer" containerID="63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"
Jul 02 13:39:50.628819 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.628799    2565 scope.go:117] "RemoveContainer" containerID="a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"
Jul 02 13:39:50.629099 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:50.629080    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55\": container with ID starting with a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55 not found: ID does not exist" containerID="a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"
Jul 02 13:39:50.629167 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.629107    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55"} err="failed to get container status \"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55\": rpc error: code = NotFound desc = could not find container \"a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55\": container with ID starting with a50c32d2d4996ce32364a14e43778fd37d90561cbf443a0d6968527d3919ec55 not found: ID does not exist"
Jul 02 13:39:50.629167 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.629125    2565 scope.go:117] "RemoveContainer" containerID="63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"
Jul 02 13:39:50.629367 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:50.629350    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528\": container with ID starting with 63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528 not found: ID does not exist" containerID="63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"
Jul 02 13:39:50.629406 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.629374    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528"} err="failed to get container status \"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528\": rpc error: code = NotFound desc = could not find container \"63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528\": container with ID starting with 63dd9905af6c245d9c3f2061e9003a5418a79264bdb28a7f64a4ed11e0dea528 not found: ID does not exist"
Jul 02 13:39:50.668603 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert\") pod \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") "
Jul 02 13:39:50.668756 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjw2\" (UniqueName: \"kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2\") pod \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") "
Jul 02 13:39:50.668756 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location\") pod \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") "
Jul 02 13:39:50.668756 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") pod \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") "
Jul 02 13:39:50.668756 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config\") pod \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\" (UID: \"c7179d3b-bfae-4193-86c5-aafc42ae7b6b\") "
Jul 02 13:39:50.669020 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.668993    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7179d3b-bfae-4193-86c5-aafc42ae7b6b" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:50.669255 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.669228    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7179d3b-bfae-4193-86c5-aafc42ae7b6b" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:50.669255 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.669238    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7179d3b-bfae-4193-86c5-aafc42ae7b6b" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:50.670818 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.670789    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2" (OuterVolumeSpecName: "kube-api-access-jzjw2") pod "c7179d3b-bfae-4193-86c5-aafc42ae7b6b" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b"). InnerVolumeSpecName "kube-api-access-jzjw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:50.670948 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.670908    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7179d3b-bfae-4193-86c5-aafc42ae7b6b" (UID: "c7179d3b-bfae-4193-86c5-aafc42ae7b6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:50.769758 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.769728    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjw2\" (UniqueName: \"kubernetes.io/projected/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-api-access-jzjw2\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:50.769758 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.769756    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:50.769975 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.769766    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:50.769975 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.769775    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:50.769975 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.769784    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7179d3b-bfae-4193-86c5-aafc42ae7b6b-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:39:50.943648 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.943560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:50.947637 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:50.947606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8b27ef-predictor-7bc7d6798-brpv5"]
Jul 02 13:39:52.707952 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:52.707910    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" path="/var/lib/kubelet/pods/c7179d3b-bfae-4193-86c5-aafc42ae7b6b/volumes"
Jul 02 13:39:54.397222 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.397184    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:54.429096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429060    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:39:54.429402 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.429492 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429404    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.429492 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429474    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.429492 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429486    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.429605 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429563    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.429605 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.429574    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7179d3b-bfae-4193-86c5-aafc42ae7b6b" containerName="storage-initializer"
Jul 02 13:39:54.434126 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.434105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.436619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.436593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dflzs\""
Jul 02 13:39:54.436619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.436617    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:39:54.436886 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.436837    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:39:54.437653 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.437630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-73b8c-predictor-serving-cert\""
Jul 02 13:39:54.437759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.437669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:39:54.437759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.437675    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:39:54.437759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.437722    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-73b8c-kube-rbac-proxy-sar-config\""
Jul 02 13:39:54.445123 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.445100    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:39:54.501107 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.501077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.501232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.501112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.501232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.501141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.501232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.501165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.501232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.501182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2djk\" (UniqueName: \"kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.601828 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.601794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.601828 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.601831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602059 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.601852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602059 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.601966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602059 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:54.601995    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-serving-cert: secret "raw-sklearn-73b8c-predictor-serving-cert" not found
Jul 02 13:39:54.602059 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.602005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2djk\" (UniqueName: \"kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602280 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:39:54.602063    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls podName:a39a274b-a2c6-47b0-9503-e098a54879ab nodeName:}" failed. No retries permitted until 2026-07-02 13:39:55.102040686 +0000 UTC m=+1500.771848033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls") pod "raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab") : secret "raw-sklearn-73b8c-predictor-serving-cert" not found
Jul 02 13:39:54.602280 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.602201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602562 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.602543    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.602619 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.602551    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.610713 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.610695    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2djk\" (UniqueName: \"kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:54.752796 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.752715    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:39:54.758214 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.758191    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:39:54.812278 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:54.812243    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:55.106273 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.106241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:55.108591 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.108572    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") pod \"raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") " pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:55.353201 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.353173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dflzs\""
Jul 02 13:39:55.361213 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.361166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:39:55.490895 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.490872    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:39:55.493402 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:39:55.493376    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39a274b_a2c6_47b0_9503_e098a54879ab.slice/crio-edc7977bf8817ef76437dec8d05658f839cf1648c15941b6a591473d14a90806 WatchSource:0}: Error finding container edc7977bf8817ef76437dec8d05658f839cf1648c15941b6a591473d14a90806: Status 404 returned error can't find the container with id edc7977bf8817ef76437dec8d05658f839cf1648c15941b6a591473d14a90806
Jul 02 13:39:55.497455 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.497434    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:39:55.637298 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.637227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerStarted","Data":"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"}
Jul 02 13:39:55.637298 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:55.637281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerStarted","Data":"edc7977bf8817ef76437dec8d05658f839cf1648c15941b6a591473d14a90806"}
Jul 02 13:39:59.654382 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:59.654347    2565 generic.go:356] "Generic (PLEG): container finished" podID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerID="0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df" exitCode=0
Jul 02 13:39:59.654770 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:39:59.654431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerDied","Data":"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"}
Jul 02 13:40:00.660911 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:00.660878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerStarted","Data":"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"}
Jul 02 13:40:00.661293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:00.660954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerStarted","Data":"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"}
Jul 02 13:40:00.661293 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:00.661141    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:40:00.680341 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:00.680296    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podStartSLOduration=3.044299404 podStartE2EDuration="6.680280397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635980993s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:00.679893413 +0000 UTC m=+1506.349700794" watchObservedRunningTime="2026-07-02 13:40:00.680280397 +0000 UTC m=+1506.350087768"
Jul 02 13:40:01.665558 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:01.665530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:40:01.666571 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:01.666543    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:02.670574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:02.670532    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:07.674982 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:07.674952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:40:07.675481 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:07.675452    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:17.675679 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:17.675631    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:24.485836 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:24.485794    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:27.676376 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:27.676331    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:37.676374 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:37.676331    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:47.675658 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:47.675616    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:40:57.676316 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:40:57.676278    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:41:07.675375 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:07.675331    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:41:15.703095 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:15.703063    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:41:24.571403 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.571362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:41:24.571806 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.571694    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" containerID="cri-o://a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8" gracePeriod=30
Jul 02 13:41:24.571806 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.571738    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kube-rbac-proxy" containerID="cri-o://cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327" gracePeriod=30
Jul 02 13:41:24.601903 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.601875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:41:24.993314 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.993233    2565 generic.go:356] "Generic (PLEG): container finished" podID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerID="cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327" exitCode=2
Jul 02 13:41:24.993447 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:24.993306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerDied","Data":"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"}
Jul 02 13:41:25.702734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:25.702693    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 13:41:27.671470 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:27.671418    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 02 13:41:29.234376 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.234353    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:41:29.325392 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config\") pod \"a39a274b-a2c6-47b0-9503-e098a54879ab\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") "
Jul 02 13:41:29.325392 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert\") pod \"a39a274b-a2c6-47b0-9503-e098a54879ab\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") "
Jul 02 13:41:29.325392 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location\") pod \"a39a274b-a2c6-47b0-9503-e098a54879ab\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") "
Jul 02 13:41:29.325680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") pod \"a39a274b-a2c6-47b0-9503-e098a54879ab\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") "
Jul 02 13:41:29.325680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325586    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2djk\" (UniqueName: \"kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk\") pod \"a39a274b-a2c6-47b0-9503-e098a54879ab\" (UID: \"a39a274b-a2c6-47b0-9503-e098a54879ab\") "
Jul 02 13:41:29.325680 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325663    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a39a274b-a2c6-47b0-9503-e098a54879ab" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:41:29.325837 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325704    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a39a274b-a2c6-47b0-9503-e098a54879ab" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:41:29.325837 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325783    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a39a274b-a2c6-47b0-9503-e098a54879ab" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:41:29.325915 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325890    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:41:29.325915 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325909    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a39a274b-a2c6-47b0-9503-e098a54879ab-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:41:29.326009 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.325919    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a39a274b-a2c6-47b0-9503-e098a54879ab-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:41:29.327393 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.327373    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a39a274b-a2c6-47b0-9503-e098a54879ab" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:41:29.327534 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.327514    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk" (OuterVolumeSpecName: "kube-api-access-w2djk") pod "a39a274b-a2c6-47b0-9503-e098a54879ab" (UID: "a39a274b-a2c6-47b0-9503-e098a54879ab"). InnerVolumeSpecName "kube-api-access-w2djk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:41:29.427234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.427196    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a39a274b-a2c6-47b0-9503-e098a54879ab-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:41:29.427234 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.427229    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2djk\" (UniqueName: \"kubernetes.io/projected/a39a274b-a2c6-47b0-9503-e098a54879ab-kube-api-access-w2djk\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:41:29.672585 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.672497    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:41:29.729260 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729224    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:41:29.729536 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729523    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container"
Jul 02 13:41:29.729579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729538    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container"
Jul 02 13:41:29.729579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729557    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="storage-initializer"
Jul 02 13:41:29.729579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729567    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="storage-initializer"
Jul 02 13:41:29.729579 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729578    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kube-rbac-proxy"
Jul 02 13:41:29.729705 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729584    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kube-rbac-proxy"
Jul 02 13:41:29.729705 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729631    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kube-rbac-proxy"
Jul 02 13:41:29.729705 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.729643    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerName="kserve-container"
Jul 02 13:41:29.732564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.732531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:41:29.733078 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.733061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.735296 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.735278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-51e73-predictor-serving-cert\""
Jul 02 13:41:29.735404 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.735314    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-51e73-kube-rbac-proxy-sar-config\""
Jul 02 13:41:29.742443 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.742419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:41:29.830598 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.830564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.830740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.830611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9f4\" (UniqueName: \"kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.830740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.830631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.830740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.830699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.830740 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.830729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9f4\" (UniqueName: \"kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931300 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931617 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931617 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931707 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931687    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.931887 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931863    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.932019 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.931930    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.933689 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.933668    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:29.939724 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:29.939702    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9f4\" (UniqueName: \"kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4\") pod \"raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:30.015341 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.015314    2565 generic.go:356] "Generic (PLEG): container finished" podID="a39a274b-a2c6-47b0-9503-e098a54879ab" containerID="a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8" exitCode=0
Jul 02 13:41:30.015503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.015372    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"
Jul 02 13:41:30.015503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.015381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerDied","Data":"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"}
Jul 02 13:41:30.015503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.015435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t" event={"ID":"a39a274b-a2c6-47b0-9503-e098a54879ab","Type":"ContainerDied","Data":"edc7977bf8817ef76437dec8d05658f839cf1648c15941b6a591473d14a90806"}
Jul 02 13:41:30.015503 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.015461    2565 scope.go:117] "RemoveContainer" containerID="cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"
Jul 02 13:41:30.024286 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.024268    2565 scope.go:117] "RemoveContainer" containerID="a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"
Jul 02 13:41:30.032264 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.032233    2565 scope.go:117] "RemoveContainer" containerID="0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"
Jul 02 13:41:30.039613 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.039509    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:41:30.039689 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.039617    2565 scope.go:117] "RemoveContainer" containerID="cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"
Jul 02 13:41:30.039906 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:41:30.039890    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327\": container with ID starting with cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327 not found: ID does not exist" containerID="cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"
Jul 02 13:41:30.039979 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.039920    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327"} err="failed to get container status \"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327\": rpc error: code = NotFound desc = could not find container \"cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327\": container with ID starting with cda579d58c156a1f9bdd72119a84945fb2f06649ff7993f09262afc00785d327 not found: ID does not exist"
Jul 02 13:41:30.039979 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.039963    2565 scope.go:117] "RemoveContainer" containerID="a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"
Jul 02 13:41:30.040203 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:41:30.040179    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8\": container with ID starting with a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8 not found: ID does not exist" containerID="a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"
Jul 02 13:41:30.040242 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.040209    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8"} err="failed to get container status \"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8\": rpc error: code = NotFound desc = could not find container \"a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8\": container with ID starting with a6aef8e45754d7896e083be668acf50813e01355e6bdb087f7b3473184a00bf8 not found: ID does not exist"
Jul 02 13:41:30.040242 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.040223    2565 scope.go:117] "RemoveContainer" containerID="0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"
Jul 02 13:41:30.040470 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:41:30.040448    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df\": container with ID starting with 0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df not found: ID does not exist" containerID="0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"
Jul 02 13:41:30.040523 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.040475    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df"} err="failed to get container status \"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df\": rpc error: code = NotFound desc = could not find container \"0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df\": container with ID starting with 0391aaa91a39979b1054105e8acad11b46f3266c1d60302282e7f46e6e3752df not found: ID does not exist"
Jul 02 13:41:30.043488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.043469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:30.044094 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.044077    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-73b8c-predictor-6d9469bbf8-cgr7t"]
Jul 02 13:41:30.172891 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.172847    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:41:30.175523 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:41:30.175493    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e0e24e_351c_4745_9fb8_59ddeb322a1c.slice/crio-a3437059160e16236e4fc9d905fcf2ea2cb74e6b0f2c9b24116c7b3c356b0de9 WatchSource:0}: Error finding container a3437059160e16236e4fc9d905fcf2ea2cb74e6b0f2c9b24116c7b3c356b0de9: Status 404 returned error can't find the container with id a3437059160e16236e4fc9d905fcf2ea2cb74e6b0f2c9b24116c7b3c356b0de9
Jul 02 13:41:30.713232 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:30.713188    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39a274b-a2c6-47b0-9503-e098a54879ab" path="/var/lib/kubelet/pods/a39a274b-a2c6-47b0-9503-e098a54879ab/volumes"
Jul 02 13:41:31.021813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:31.021725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerStarted","Data":"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"}
Jul 02 13:41:31.021813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:31.021786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerStarted","Data":"a3437059160e16236e4fc9d905fcf2ea2cb74e6b0f2c9b24116c7b3c356b0de9"}
Jul 02 13:41:34.034684 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:34.034649    2565 generic.go:356] "Generic (PLEG): container finished" podID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerID="6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0" exitCode=0
Jul 02 13:41:34.035149 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:34.034722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerDied","Data":"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"}
Jul 02 13:41:35.040563 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:35.040531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerStarted","Data":"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"}
Jul 02 13:41:35.040927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:35.040589    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerStarted","Data":"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"}
Jul 02 13:41:35.040927 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:35.040781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:35.059568 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:35.059524    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podStartSLOduration=2.426362907 podStartE2EDuration="6.059509359s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633146452s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:41:35.057841701 +0000 UTC m=+1600.727649069" watchObservedRunningTime="2026-07-02 13:41:35.059509359 +0000 UTC m=+1600.729316727"
Jul 02 13:41:36.044537 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:36.044507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:36.045734 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:36.045705    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:41:37.048382 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:37.048340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:41:42.052697 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:42.052670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:41:42.053199 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:42.053171    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:41:52.053644 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:41:52.053559    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:02.053516 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:02.053469    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:12.053506 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:12.053464    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:22.053566 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:22.053530    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:32.053911 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:32.053873    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:42.053170 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:42.053130    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:42:52.053743 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:52.053716    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:42:59.839457 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:59.839424    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:42:59.839902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:59.839721    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" containerID="cri-o://ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f" gracePeriod=30
Jul 02 13:42:59.839902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:59.839791    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kube-rbac-proxy" containerID="cri-o://79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778" gracePeriod=30
Jul 02 13:42:59.870138 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:42:59.870115    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:00.363306 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:00.363275    2565 generic.go:356] "Generic (PLEG): container finished" podID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerID="79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778" exitCode=2
Jul 02 13:43:00.363488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:00.363326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerDied","Data":"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"}
Jul 02 13:43:02.049147 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:02.049112    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 02 13:43:02.053458 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:02.053434    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 13:43:04.186043 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.186020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:43:04.240779 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.240713    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config\") pod \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") "
Jul 02 13:43:04.240779 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.240746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert\") pod \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") "
Jul 02 13:43:04.240779 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.240766    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p9f4\" (UniqueName: \"kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4\") pod \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") "
Jul 02 13:43:04.241057 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.240798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location\") pod \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") "
Jul 02 13:43:04.241057 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.240863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls\") pod \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\" (UID: \"60e0e24e-351c-4745-9fb8-59ddeb322a1c\") "
Jul 02 13:43:04.241209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.241133    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60e0e24e-351c-4745-9fb8-59ddeb322a1c" (UID: "60e0e24e-351c-4745-9fb8-59ddeb322a1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:04.241306 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.241205    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60e0e24e-351c-4745-9fb8-59ddeb322a1c" (UID: "60e0e24e-351c-4745-9fb8-59ddeb322a1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:43:04.241347 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.241300    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60e0e24e-351c-4745-9fb8-59ddeb322a1c" (UID: "60e0e24e-351c-4745-9fb8-59ddeb322a1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:43:04.242813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.242790    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60e0e24e-351c-4745-9fb8-59ddeb322a1c" (UID: "60e0e24e-351c-4745-9fb8-59ddeb322a1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:43:04.242813 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.242791    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4" (OuterVolumeSpecName: "kube-api-access-7p9f4") pod "60e0e24e-351c-4745-9fb8-59ddeb322a1c" (UID: "60e0e24e-351c-4745-9fb8-59ddeb322a1c"). InnerVolumeSpecName "kube-api-access-7p9f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:43:04.341997 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.341977    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-cabundle-cert\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:04.341997 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.341996    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p9f4\" (UniqueName: \"kubernetes.io/projected/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-api-access-7p9f4\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:04.342120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.342007    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kserve-provision-location\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:04.342120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.342016    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60e0e24e-351c-4745-9fb8-59ddeb322a1c-proxy-tls\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:04.342120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.342023    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60e0e24e-351c-4745-9fb8-59ddeb322a1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:04.379994 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.379972    2565 generic.go:356] "Generic (PLEG): container finished" podID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerID="ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f" exitCode=0
Jul 02 13:43:04.380080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.380011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerDied","Data":"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"}
Jul 02 13:43:04.380080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.380026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"
Jul 02 13:43:04.380080 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.380058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm" event={"ID":"60e0e24e-351c-4745-9fb8-59ddeb322a1c","Type":"ContainerDied","Data":"a3437059160e16236e4fc9d905fcf2ea2cb74e6b0f2c9b24116c7b3c356b0de9"}
Jul 02 13:43:04.380189 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.380086    2565 scope.go:117] "RemoveContainer" containerID="79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"
Jul 02 13:43:04.388601 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.388585    2565 scope.go:117] "RemoveContainer" containerID="ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"
Jul 02 13:43:04.395721 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.395702    2565 scope.go:117] "RemoveContainer" containerID="6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"
Jul 02 13:43:04.402649 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.402626    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:43:04.403319 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.403304    2565 scope.go:117] "RemoveContainer" containerID="79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"
Jul 02 13:43:04.403570 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:04.403552    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778\": container with ID starting with 79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778 not found: ID does not exist" containerID="79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"
Jul 02 13:43:04.403609 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.403578    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778"} err="failed to get container status \"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778\": rpc error: code = NotFound desc = could not find container \"79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778\": container with ID starting with 79e8f1b0094b3ae0a8a3a454423a0ac4b861921d96858f3584a7eb7684029778 not found: ID does not exist"
Jul 02 13:43:04.403609 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.403594    2565 scope.go:117] "RemoveContainer" containerID="ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"
Jul 02 13:43:04.403837 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:04.403822    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f\": container with ID starting with ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f not found: ID does not exist" containerID="ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"
Jul 02 13:43:04.403873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.403844    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f"} err="failed to get container status \"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f\": rpc error: code = NotFound desc = could not find container \"ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f\": container with ID starting with ede2b9e4ff16329f96dd895d8760fec7591260e222ffa67d02acda44d25e778f not found: ID does not exist"
Jul 02 13:43:04.403873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.403859    2565 scope.go:117] "RemoveContainer" containerID="6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"
Jul 02 13:43:04.404211 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:04.404185    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0\": container with ID starting with 6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0 not found: ID does not exist" containerID="6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"
Jul 02 13:43:04.404439 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.404218    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0"} err="failed to get container status \"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0\": rpc error: code = NotFound desc = could not find container \"6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0\": container with ID starting with 6fd96d00ce695d65c384780afd74ad93d908407384a6000310c7652dd8b265c0 not found: ID does not exist"
Jul 02 13:43:04.406482 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.406459    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51e73-predictor-86dc58d6-flpzm"]
Jul 02 13:43:04.707239 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:04.707204    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" path="/var/lib/kubelet/pods/60e0e24e-351c-4745-9fb8-59ddeb322a1c/volumes"
Jul 02 13:43:05.782370 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cwsqs/must-gather-vb456"]
Jul 02 13:43:05.782752 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782718    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container"
Jul 02 13:43:05.782752 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782733    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container"
Jul 02 13:43:05.782752 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="storage-initializer"
Jul 02 13:43:05.782752 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="storage-initializer"
Jul 02 13:43:05.782902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kube-rbac-proxy"
Jul 02 13:43:05.782902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782766    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kube-rbac-proxy"
Jul 02 13:43:05.782902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782808    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kube-rbac-proxy"
Jul 02 13:43:05.782902 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.782819    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e0e24e-351c-4745-9fb8-59ddeb322a1c" containerName="kserve-container"
Jul 02 13:43:05.786035 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.786004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:05.787220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.787206    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.789432 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.789415    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cwsqs\"/\"default-dockercfg-xnmm7\""
Jul 02 13:43:05.789543 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.789433    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwsqs\"/\"kube-root-ca.crt\""
Jul 02 13:43:05.790574 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.790555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwsqs\"/\"openshift-service-ca.crt\""
Jul 02 13:43:05.792873 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.792854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwsqs/must-gather-vb456"]
Jul 02 13:43:05.854171 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.854147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.854292 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.854206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzzf\" (UniqueName: \"kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.887220 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.887189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:05.955398 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.955358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzzf\" (UniqueName: \"kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.955522 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.955457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.955759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.955742    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:05.964949 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:05.964911    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzzf\" (UniqueName: \"kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf\") pod \"must-gather-vb456\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") " pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:06.111281 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:06.111223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:06.232605 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:06.232580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwsqs/must-gather-vb456"]
Jul 02 13:43:06.234608 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:43:06.234577    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0013e26b_afb4_4792_8ffd_ae49e51fa070.slice/crio-7fb06642a093c3c2454050351bfafb88e186635998aa960539d7c341dded7e52 WatchSource:0}: Error finding container 7fb06642a093c3c2454050351bfafb88e186635998aa960539d7c341dded7e52: Status 404 returned error can't find the container with id 7fb06642a093c3c2454050351bfafb88e186635998aa960539d7c341dded7e52
Jul 02 13:43:06.236373 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:06.236355    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:43:06.389711 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:06.389645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwsqs/must-gather-vb456" event={"ID":"0013e26b-afb4-4792-8ffd-ae49e51fa070","Type":"ContainerStarted","Data":"7fb06642a093c3c2454050351bfafb88e186635998aa960539d7c341dded7e52"}
Jul 02 13:43:13.424606 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:13.424572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwsqs/must-gather-vb456" event={"ID":"0013e26b-afb4-4792-8ffd-ae49e51fa070","Type":"ContainerStarted","Data":"7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81"}
Jul 02 13:43:13.424994 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:13.424615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwsqs/must-gather-vb456" event={"ID":"0013e26b-afb4-4792-8ffd-ae49e51fa070","Type":"ContainerStarted","Data":"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"}
Jul 02 13:43:13.441776 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:13.441723    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cwsqs/must-gather-vb456" podStartSLOduration=2.370738679 podStartE2EDuration="8.441707159s" totalImagesPullingTime="6.07096848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:43:13.439485044 +0000 UTC m=+1699.109292412" watchObservedRunningTime="2026-07-02 13:43:13.441707159 +0000 UTC m=+1699.111514527"
Jul 02 13:43:34.508451 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:34.508417    2565 generic.go:356] "Generic (PLEG): container finished" podID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerID="2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac" exitCode=0
Jul 02 13:43:34.508869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:34.508487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwsqs/must-gather-vb456" event={"ID":"0013e26b-afb4-4792-8ffd-ae49e51fa070","Type":"ContainerDied","Data":"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"}
Jul 02 13:43:34.508869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:34.508801    2565 scope.go:117] "RemoveContainer" containerID="2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"
Jul 02 13:43:34.672298 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:34.672271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cwsqs_must-gather-vb456_0013e26b-afb4-4792-8ffd-ae49e51fa070/gather/0.log"
Jul 02 13:43:35.071182 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:35.071152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:35.237383 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:35.237304    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:35.336346 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:35.336303    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:37.937344 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:37.937312    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5d8ms_38b4b3cd-994a-47be-a765-884a6af449c8/global-pull-secret-syncer/0.log"
Jul 02 13:43:38.104199 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:38.104169    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log"
Jul 02 13:43:38.255002 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:38.254914    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-102.ec2.internal_67cd314639f46c104191661c02cd9b02/haproxy/0.log"
Jul 02 13:43:40.054559 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.054526    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-cwsqs/must-gather-vb456"]
Jul 02 13:43:40.055031 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.054688    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-cwsqs/must-gather-vb456" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="copy" containerID="cri-o://7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81" gracePeriod=2
Jul 02 13:43:40.059431 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.059378    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-cwsqs/must-gather-vb456"]
Jul 02 13:43:40.285092 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.285068    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cwsqs_must-gather-vb456_0013e26b-afb4-4792-8ffd-ae49e51fa070/copy/0.log"
Jul 02 13:43:40.287699 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.287681    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:40.289643 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:40.289619    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vb456\" is forbidden: User \"system:node:ip-10-0-134-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cwsqs\": no relationship found between node 'ip-10-0-134-102.ec2.internal' and this object" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:40.453532 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.453475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output\") pod \"0013e26b-afb4-4792-8ffd-ae49e51fa070\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") "
Jul 02 13:43:40.453532 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.453508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xzzf\" (UniqueName: \"kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf\") pod \"0013e26b-afb4-4792-8ffd-ae49e51fa070\" (UID: \"0013e26b-afb4-4792-8ffd-ae49e51fa070\") "
Jul 02 13:43:40.454852 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.454828    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0013e26b-afb4-4792-8ffd-ae49e51fa070" (UID: "0013e26b-afb4-4792-8ffd-ae49e51fa070"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:43:40.455570 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.455549    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf" (OuterVolumeSpecName: "kube-api-access-7xzzf") pod "0013e26b-afb4-4792-8ffd-ae49e51fa070" (UID: "0013e26b-afb4-4792-8ffd-ae49e51fa070"). InnerVolumeSpecName "kube-api-access-7xzzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:43:40.531603 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.531583    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cwsqs_must-gather-vb456_0013e26b-afb4-4792-8ffd-ae49e51fa070/copy/0.log"
Jul 02 13:43:40.531926 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.531907    2565 generic.go:356] "Generic (PLEG): container finished" podID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerID="7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81" exitCode=143
Jul 02 13:43:40.532016 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.531977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:40.532075 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.532013    2565 scope.go:117] "RemoveContainer" containerID="7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81"
Jul 02 13:43:40.534244 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:40.534210    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vb456\" is forbidden: User \"system:node:ip-10-0-134-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cwsqs\": no relationship found between node 'ip-10-0-134-102.ec2.internal' and this object" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:40.539972 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.539929    2565 scope.go:117] "RemoveContainer" containerID="2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"
Jul 02 13:43:40.542130 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:40.542104    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vb456\" is forbidden: User \"system:node:ip-10-0-134-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cwsqs\": no relationship found between node 'ip-10-0-134-102.ec2.internal' and this object" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" pod="openshift-must-gather-cwsqs/must-gather-vb456"
Jul 02 13:43:40.552919 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.552902    2565 scope.go:117] "RemoveContainer" containerID="7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81"
Jul 02 13:43:40.553198 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:40.553180    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81\": container with ID starting with 7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81 not found: ID does not exist" containerID="7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81"
Jul 02 13:43:40.553250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.553207    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81"} err="failed to get container status \"7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81\": rpc error: code = NotFound desc = could not find container \"7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81\": container with ID starting with 7b2c60a478d996b5f8d0c0c93cf98cf7efcde1e7eef575a3bc5a97190b0e7f81 not found: ID does not exist"
Jul 02 13:43:40.553250 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.553224    2565 scope.go:117] "RemoveContainer" containerID="2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"
Jul 02 13:43:40.553441 ip-10-0-134-102 kubenswrapper[2565]: E0702 13:43:40.553425    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac\": container with ID starting with 2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac not found: ID does not exist" containerID="2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"
Jul 02 13:43:40.553484 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.553447    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac"} err="failed to get container status \"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac\": rpc error: code = NotFound desc = could not find container \"2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac\": container with ID starting with 2dd554fd28a33e38abff2ef2ee51780b23025c53da75b5b2609b1b9da659dcac not found: ID does not exist"
Jul 02 13:43:40.554745 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.554731    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzzf\" (UniqueName: \"kubernetes.io/projected/0013e26b-afb4-4792-8ffd-ae49e51fa070-kube-api-access-7xzzf\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:40.554788 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.554747    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0013e26b-afb4-4792-8ffd-ae49e51fa070-must-gather-output\") on node \"ip-10-0-134-102.ec2.internal\" DevicePath \"\""
Jul 02 13:43:40.707551 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.707468    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" path="/var/lib/kubelet/pods/0013e26b-afb4-4792-8ffd-ae49e51fa070/volumes"
Jul 02 13:43:40.940127 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.940092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:40.993350 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:40.993270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4scd4_ef581d1d-f306-4d3a-b07f-68a2fb289e78/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:42.601991 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:42.601967    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-vm8xz_a639313a-e342-4393-a017-c9ee4dd9da02/cluster-monitoring-operator/0.log"
Jul 02 13:43:42.630628 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:42.630594    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-w7lzm_d1e83c05-f5fc-4f05-92f3-0e41c86fc624/kube-state-metrics/0.log"
Jul 02 13:43:42.653262 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:42.653240    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-w7lzm_d1e83c05-f5fc-4f05-92f3-0e41c86fc624/kube-rbac-proxy-main/0.log"
Jul 02 13:43:42.680514 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:42.680488    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-w7lzm_d1e83c05-f5fc-4f05-92f3-0e41c86fc624/kube-rbac-proxy-self/0.log"
Jul 02 13:43:42.982079 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:42.982045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdbs7_eba184bd-079b-4132-9470-69224d9eed57/node-exporter/0.log"
Jul 02 13:43:43.006881 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:43.006856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdbs7_eba184bd-079b-4132-9470-69224d9eed57/kube-rbac-proxy/0.log"
Jul 02 13:43:43.033855 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:43.033836    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdbs7_eba184bd-079b-4132-9470-69224d9eed57/init-textfile/0.log"
Jul 02 13:43:43.064957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:43.064919    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2lrbn_e88fec06-a219-4b2e-8e56-f71a07250ee1/kube-rbac-proxy-main/0.log"
Jul 02 13:43:43.090787 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:43.090754    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2lrbn_e88fec06-a219-4b2e-8e56-f71a07250ee1/kube-rbac-proxy-self/0.log"
Jul 02 13:43:43.117752 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:43.117729    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-2lrbn_e88fec06-a219-4b2e-8e56-f71a07250ee1/openshift-state-metrics/0.log"
Jul 02 13:43:45.540421 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:45.540386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/2.log"
Jul 02 13:43:45.551747 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:45.551714    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-6pm9w_07174aea-ad00-466e-a362-cdd03e3c9c1c/console-operator/3.log"
Jul 02 13:43:45.992694 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:45.992621    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78b97bdf87-nbnkk_5c1fe43d-d733-4a6b-958d-840c0ab14535/console/0.log"
Jul 02 13:43:46.645099 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"]
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="copy"
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645371    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="copy"
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645381    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="gather"
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645386    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="gather"
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645445    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="copy"
Jul 02 13:43:46.645488 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.645452    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0013e26b-afb4-4792-8ffd-ae49e51fa070" containerName="gather"
Jul 02 13:43:46.650253 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.650232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.652546 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.652527    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ckq4l\"/\"kube-root-ca.crt\""
Jul 02 13:43:46.652546 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.652539    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ckq4l\"/\"openshift-service-ca.crt\""
Jul 02 13:43:46.652696 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.652639    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ckq4l\"/\"default-dockercfg-hw4ls\""
Jul 02 13:43:46.657824 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.657803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"]
Jul 02 13:43:46.698953 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.698888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-sys\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.699096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.698967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6cr\" (UniqueName: \"kubernetes.io/projected/c41cfb82-3d27-42ab-84bb-346ea70fa923-kube-api-access-xf6cr\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.699096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.699009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-proc\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.699096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.699027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-podres\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.699096 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.699088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-lib-modules\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-proc\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799759 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-podres\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-lib-modules\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-sys\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6cr\" (UniqueName: \"kubernetes.io/projected/c41cfb82-3d27-42ab-84bb-346ea70fa923-kube-api-access-xf6cr\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799900    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-lib-modules\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.799957 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799901    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-podres\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.800120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799956    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-sys\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.800120 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.799968    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c41cfb82-3d27-42ab-84bb-346ea70fa923-proc\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.808329 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.808301    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6cr\" (UniqueName: \"kubernetes.io/projected/c41cfb82-3d27-42ab-84bb-346ea70fa923-kube-api-access-xf6cr\") pod \"perf-node-gather-daemonset-ct994\" (UID: \"c41cfb82-3d27-42ab-84bb-346ea70fa923\") " pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:46.960659 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:46.960595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:47.085920 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.085897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"]
Jul 02 13:43:47.087232 ip-10-0-134-102 kubenswrapper[2565]: W0702 13:43:47.087205    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc41cfb82_3d27_42ab_84bb_346ea70fa923.slice/crio-46781f824b860603c19d1e1b3dae7e3d12e035f28a81b490a9b9eeb9eb81421b WatchSource:0}: Error finding container 46781f824b860603c19d1e1b3dae7e3d12e035f28a81b490a9b9eeb9eb81421b: Status 404 returned error can't find the container with id 46781f824b860603c19d1e1b3dae7e3d12e035f28a81b490a9b9eeb9eb81421b
Jul 02 13:43:47.563186 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.563152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994" event={"ID":"c41cfb82-3d27-42ab-84bb-346ea70fa923","Type":"ContainerStarted","Data":"4fd7e45434466268678decc322af01fc5a83a235e5b8858aadd0c7d9083adfbe"}
Jul 02 13:43:47.563348 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.563204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994" event={"ID":"c41cfb82-3d27-42ab-84bb-346ea70fa923","Type":"ContainerStarted","Data":"46781f824b860603c19d1e1b3dae7e3d12e035f28a81b490a9b9eeb9eb81421b"}
Jul 02 13:43:47.563348 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.563273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:47.600193 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.600152    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994" podStartSLOduration=1.600139806 podStartE2EDuration="1.600139806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:43:47.59904885 +0000 UTC m=+1733.268856221" watchObservedRunningTime="2026-07-02 13:43:47.600139806 +0000 UTC m=+1733.269947173"
Jul 02 13:43:47.635422 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.635404    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4d98s_c2d57480-bc69-4de0-954a-098c7f6e0903/dns/0.log"
Jul 02 13:43:47.663531 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.663503    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4d98s_c2d57480-bc69-4de0-954a-098c7f6e0903/kube-rbac-proxy/0.log"
Jul 02 13:43:47.860302 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:47.860222    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v295b_91893a1e-ad7e-45b5-a53c-17c189ce5221/dns-node-resolver/0.log"
Jul 02 13:43:48.357106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:48.357073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85d4f8f779-ncqr4_b6151020-9cdd-4b7a-a544-60dcc396d622/registry/0.log"
Jul 02 13:43:48.431106 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:48.431077    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rv7sq_8f0cc46a-0ad9-4952-bf16-b87d344e38ee/node-ca/0.log"
Jul 02 13:43:49.293399 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:49.293372    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f7cb747c-qh9k6_e0f775f1-3efe-4432-b60e-f184557e7f60/router/0.log"
Jul 02 13:43:49.726649 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:49.726574    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vm7c9_b1311dc9-756f-41d6-8749-bfdbd03fe4cd/serve-healthcheck-canary/0.log"
Jul 02 13:43:50.285268 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:50.285238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7plw_71251c71-6232-4757-8831-5794576e5a2a/kube-rbac-proxy/0.log"
Jul 02 13:43:50.308005 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:50.307989    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7plw_71251c71-6232-4757-8831-5794576e5a2a/exporter/0.log"
Jul 02 13:43:50.332181 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:50.332157    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7plw_71251c71-6232-4757-8831-5794576e5a2a/extractor/0.log"
Jul 02 13:43:52.425971 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:52.425905    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-694f5b7594-4fd99_8edec11c-29ad-4026-ab15-f1e3ca9af83d/manager/0.log"
Jul 02 13:43:52.630494 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:52.630462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-xgsrx_0ad4d8b6-4a3e-4d6a-a0be-e92b2c4aec3f/manager/0.log"
Jul 02 13:43:52.673746 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:52.673716    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6k65s_e7eced54-ff41-4c47-bf44-e6c40c8b0fc4/s3-init/0.log"
Jul 02 13:43:52.691485 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:52.691417    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6k65s_e7eced54-ff41-4c47-bf44-e6c40c8b0fc4/download-hf-model/0.log"
Jul 02 13:43:52.723286 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:52.723261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-9f7lb_d0882121-7575-4ae3-be86-e7762fba257c/seaweedfs/0.log"
Jul 02 13:43:53.576967 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:53.576930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ckq4l/perf-node-gather-daemonset-ct994"
Jul 02 13:43:58.425877 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.425849    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9tszt_0a94fc30-e47f-488d-9c59-8e7fc264baa9/kube-multus/0.log"
Jul 02 13:43:58.646401 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.646375    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/kube-multus-additional-cni-plugins/0.log"
Jul 02 13:43:58.669810 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.669786    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/egress-router-binary-copy/0.log"
Jul 02 13:43:58.694954 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.694917    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/cni-plugins/0.log"
Jul 02 13:43:58.719303 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.719282    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/bond-cni-plugin/0.log"
Jul 02 13:43:58.742698 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.742671    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/routeoverride-cni/0.log"
Jul 02 13:43:58.766809 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.766787    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/whereabouts-cni-bincopy/0.log"
Jul 02 13:43:58.790033 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:58.790007    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4dp4_45afd23b-217f-47d3-85bc-0ae6e6390ece/whereabouts-cni/0.log"
Jul 02 13:43:59.111968 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:59.111923    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ffcp9_206cd077-8764-42f9-86ca-71d3cc81fa57/network-metrics-daemon/0.log"
Jul 02 13:43:59.134869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:43:59.134839    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ffcp9_206cd077-8764-42f9-86ca-71d3cc81fa57/kube-rbac-proxy/0.log"
Jul 02 13:44:00.161564 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.161538    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/ovn-controller/0.log"
Jul 02 13:44:00.179598 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.179568    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/ovn-acl-logging/0.log"
Jul 02 13:44:00.208950 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.208913    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/kube-rbac-proxy-node/0.log"
Jul 02 13:44:00.233587 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.233558    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 13:44:00.254209 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.254189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/northd/0.log"
Jul 02 13:44:00.276444 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.276425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/nbdb/0.log"
Jul 02 13:44:00.300478 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.300434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/sbdb/0.log"
Jul 02 13:44:00.364797 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:00.364760    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mprzv_af575503-ae25-49d2-8280-39c4d2d6fe16/ovnkube-controller/0.log"
Jul 02 13:44:01.601609 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:01.601572    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-55ttw_9f73406e-e073-47bb-b010-315f56ab60d8/network-check-target-container/0.log"
Jul 02 13:44:02.693869 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:02.693798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-t5nw2_e9039d09-c0b2-443f-bf7f-0c77b104026b/iptables-alerter/0.log"
Jul 02 13:44:03.357974 ip-10-0-134-102 kubenswrapper[2565]: I0702 13:44:03.357930    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dhsvd_6f926e87-2b39-40ae-9798-072eba46899d/tuned/0.log"