Jul 06 20:32:59.356176 ip-10-0-139-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 20:32:59.356188 ip-10-0-139-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 20:32:59.356198 ip-10-0-139-250 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 20:32:59.356480 ip-10-0-139-250 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 20:33:09.572888 ip-10-0-139-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 20:33:09.572908 ip-10-0-139-250 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a17a312f396241d090a3f75e4e2a0d60 --
Jul 06 20:35:03.968571 ip-10-0-139-250 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:35:04.428400 ip-10-0-139-250 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:04.428400 ip-10-0-139-250 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:35:04.428400 ip-10-0-139-250 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:04.428400 ip-10-0-139-250 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:35:04.428400 ip-10-0-139-250 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:04.429325 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.429217    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:35:04.434471 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434442    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.434471 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434466    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.434471 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434470    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.434471 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434473    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.434471 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434479    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434483    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434486    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434489    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434492    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434494    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434496    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434500    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434504    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434507    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434509    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434512    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434514    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434516    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434519    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434522    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434525    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434527    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434530    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434533    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.434687 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434535    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434538    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434540    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434543    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434545    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434547    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434549    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434552    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434554    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434556    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434559    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434561    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434563    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434566    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434569    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434573    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434576    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434579    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434582    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.435170 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434585    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434588    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434590    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434593    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434595    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434598    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434600    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434604    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434606    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434608    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434611    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434613    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434616    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434618    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434621    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434624    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434626    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434628    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434631    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434633    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.435670 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434636    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434638    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434640    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434643    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434645    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434647    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434650    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434653    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434655    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434657    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434660    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434662    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434664    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434667    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434669    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434673    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434676    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434678    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434680    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434683    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.436138 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434686    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434688    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434691    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434694    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434697    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434700    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434702    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434704    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434707    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434710    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434712    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434715    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434717    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434719    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434722    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.434724    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435465    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435473    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435477    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.436627 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435481    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435484    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435486    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435489    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435491    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435494    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435496    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435499    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435502    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435504    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435506    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435509    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435511    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435514    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435517    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435519    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435522    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435524    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435527    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435530    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.437074 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435532    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435535    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435537    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435539    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435542    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435544    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435547    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435549    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435551    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435554    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435556    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435559    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435561    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435564    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435566    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435569    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435572    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435575    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435577    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435580    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.437582 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435582    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435585    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435587    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435589    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435592    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435594    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435596    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435599    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435605    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435608    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435611    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435614    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435618    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435621    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435624    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435629    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435632    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435635    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.438075 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435638    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435641    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435643    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435646    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435649    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435651    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435654    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435656    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435660    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435662    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435665    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435667    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435670    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435672    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435675    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435677    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435680    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435682    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435685    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435687    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.438527 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435690    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435692    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435696    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435698    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435701    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435703    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435705    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435708    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435710    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435713    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435716    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435718    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435720    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435723    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435726    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435728    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435731    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.435733    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436123    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436127    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.438991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436130    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436132    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436134    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436137    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436141    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436145    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436148    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436151    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436154    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436156    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436159    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436161    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436164    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436167    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436171    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436174    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436176    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436179    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436181    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436183    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.439586 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436186    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436188    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436191    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436194    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436197    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436199    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436201    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436204    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436206    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436209    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436211    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436214    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436216    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436218    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436221    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436223    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436225    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436228    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436231    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436233    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.440058 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436235    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436238    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436240    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436243    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436245    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436247    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436264    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436275    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436277    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436280    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436283    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436285    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436287    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436292    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436295    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436298    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436301    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436303    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436306    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436308    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.440588 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436311    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436313    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436316    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436318    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436320    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436323    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436325    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436329    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436334    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436337    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436341    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436343    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436346    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436351    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436353    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436356    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436358    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436361    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436363    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.441069 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436366    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436369    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436371    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436374    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436376    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436378    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436381    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436384    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436386    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436388    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436391    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436394    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436396    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436398    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436401    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436403    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436405    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.436408    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436487    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436503    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436513    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:35:04.441542 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436521    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436528    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436533    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436537    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436542    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436545    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436547    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436552    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436555    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436557    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436560    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436563    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436566    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436569    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436574    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436578    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436581    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436584    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436586    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436590    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436593    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436596    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436599    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:35:04.442047 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436602    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436605    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436608    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436610    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436613    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436616    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436620    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436623    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436626    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436630    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436633    2569 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436636    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436641    2569 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436644    2569 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436646    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436650    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436652    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436658    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436661    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436664    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436667    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436670    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436672    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436677    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436680    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:35:04.442608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436682    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436685    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436688    2569 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436691    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436694    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436697    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436700    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436703    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436706    2569 flags.go:64] FLAG: --help="false"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436708    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436711    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436714    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436716    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436719    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436723    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436726    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436730    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436732    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436735    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436737    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436740    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436743    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436746    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436748    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:35:04.443210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436751    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436753    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436756    2569 flags.go:64] FLAG: --lock-file=""
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436759    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436761    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436764    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436770    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436774    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436776    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436779    2569 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436781    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436784    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436787    2569 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436790    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436794    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436797    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436801    2569 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436803    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436806    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436809    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436811    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436814    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436817    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436820    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436829    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:35:04.443830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436831    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436834    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436837    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436840    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436846    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436849    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436852    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436854    2569 flags.go:64] FLAG: --port="10250"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436857    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436860    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00add252591612f98"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436863    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436865    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436868    2569 flags.go:64] FLAG: --register-node="true"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436871    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436876    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436879    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436882    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436884    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436888    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436891    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436893    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436896    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436899    2569 flags.go:64] FLAG: --runonce="false"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436901    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436904    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:35:04.444442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436907    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436909    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436912    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436914    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436917    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436920    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436924    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436927    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436930    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436933    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436935    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436938    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436941    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436946    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436950    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436952    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436956    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436958    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436961    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436963    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436966    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436970    2569 flags.go:64] FLAG: --v="2"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436975    2569 flags.go:64] FLAG: --version="false"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436979    2569 flags.go:64] FLAG: --vmodule=""
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436983    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:35:04.445044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.436986    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437103    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437107    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437109    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437111    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437114    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437116    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437119    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437121    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437123    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437125    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437128    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437130    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437133    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437136    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437138    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437141    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437143    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437146    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437148    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.445647 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437151    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437153    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437155    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437158    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437160    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437163    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437166    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437168    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437173    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437175    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437178    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437181    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437183    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437185    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437187    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437190    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437193    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437196    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437200    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.446130 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437203    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437205    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437208    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437210    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437213    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437215    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437219    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437222    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437225    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437228    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437231    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437234    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437237    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437239    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437242    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437244    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437247    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437263    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437266    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.446599 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437268    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437271    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437276    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437279    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437281    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437284    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437286    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437289    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437291    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437294    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437296    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437299    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437301    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437303    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437306    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437308    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437310    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437313    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437315    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437318    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.447049 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437320    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437322    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437325    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437327    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437329    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437332    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437334    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437337    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437339    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437341    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437343    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437346    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437348    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437350    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437354    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437357    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437360    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437362    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437364    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.447543 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437367    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437370    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.437372    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.437911    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.445352    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.445371    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.445385    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.445394    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445453    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445458    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445461    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445464    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445467    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445470    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445473    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.447991 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445476    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445478    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445480    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445483    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445486    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445488    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445491    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445496    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445499    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445503    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445505    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445508    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445511    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445514    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445516    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445519    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445522    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445524    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445527    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.448372 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445529    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445532    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445534    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445538    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445540    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445543    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445545    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445548    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445550    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445552    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445555    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445557    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445559    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445562    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445564    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445567    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445569    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445571    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445574    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445577    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.448816 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445579    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445582    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445584    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445587    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445589    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445591    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445594    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445596    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445598    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445601    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445603    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445606    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445609    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445611    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445614    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445617    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445620    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445622    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445625    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.449335 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445627    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445630    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445632    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445634    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445636    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445639    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445641    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445644    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445646    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445648    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445652    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445656    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445659    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445661    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445664    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445666    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445669    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445671    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445674    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.449789 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445676    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445678    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445681    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445683    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445686    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445688    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445691    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445694    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445696    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445699    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445702    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445704    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445706    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445709    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445712    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.450237 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.445718    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445847    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445852    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445855    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445858    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445862    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445866    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445869    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445871    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445874    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445876    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445879    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445881    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445883    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445886    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445888    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445890    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445893    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445895    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445898    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:04.450612 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445900    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445902    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445904    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445907    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445910    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445912    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445915    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445917    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445920    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445923    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445925    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445928    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445930    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445932    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445934    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445937    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445939    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445941    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445944    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:04.451080 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445946    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445948    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445951    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445953    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445955    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445958    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445962    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445965    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445968    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445970    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445973    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445975    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445978    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445980    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445982    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445985    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445988    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445990    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445993    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:04.451549 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.445996    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446007    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446011    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446013    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446017    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446019    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446022    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446024    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446027    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446029    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446032    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446034    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446037    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446039    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446041    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446044    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446046    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446048    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446051    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446053    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:04.451995 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446056    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446058    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446061    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446063    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446065    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446068    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446070    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446072    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446074    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446077    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446079    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446083    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446085    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446088    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446090    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446092    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446095    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446098    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446100    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446103    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:04.452469 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446105    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:04.446107    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.446112    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.446239    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.448171    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.449550    2569 server.go:1019] "Starting client certificate rotation"
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.449652    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.449688    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:04.452968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.452112    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:04.455667 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.455641    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:04.468042 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.468014    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:35:04.470492 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.470476    2569 log.go:25] "Validated CRI v1 image API"
Jul 06 20:35:04.471872 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.471843    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:35:04.474799 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.474776    2569 fs.go:135] Filesystem UUIDs: map[2f34bba1-cb66-42e4-b026-7078a5bffcf7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 98b8967e-9a2c-46f4-9cff-1b479708da00:/dev/nvme0n1p3]
Jul 06 20:35:04.474904 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.474795    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:35:04.477224 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.477203    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:04.480210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.480091    2569 manager.go:216] Machine: {Timestamp:2026-07-06 20:35:04.478491286 +0000 UTC m=+0.371809790 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200512 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261dcb6adcd37e515082a8644eb5ea SystemUUID:ec261dcb-6adc-d37e-5150-82a8644eb5ea BootID:a17a312f-3962-41d0-90a3-f75e4e2a0d60 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:2a:f9:69:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:2a:f9:69:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:cf:69:14:bc:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:35:04.480210 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.480205    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:35:04.480338 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.480307    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:35:04.483951 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.483920    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:35:04.484088 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.483955    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:35:04.484650 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.484639    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:35:04.484685 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.484651    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:35:04.484685 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.484659    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:04.486015 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.485996    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:04.487176 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.487164    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:35:04.490096 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490084    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:04.490216 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490207    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:35:04.490248 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490241    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:35:04.490290 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490275    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:35:04.490316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490293    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:35:04.490316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490307    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:35:04.490384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490320    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:35:04.490785 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490772    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:04.490830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.490791    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:04.494020 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.493996    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:35:04.495633 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.495618    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:35:04.495703 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.495648    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:35:04.497011 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497000    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497015    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497022    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497027    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497035    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497042    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497050    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:35:04.497054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497054    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:35:04.497219 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497061    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:35:04.497219 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497077    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:35:04.497219 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497090    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:35:04.497219 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497096    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:35:04.497993 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497984    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:35:04.497993 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.497993    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:35:04.500821 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.500796    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:35:04.501019 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.500982    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:35:04.501092 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.500995    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:35:04.501997 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.501979    2569 server.go:1268] "Started kubelet"
Jul 06 20:35:04.502088 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.502058    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:35:04.502176 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.502125    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:35:04.502221 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.502201    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:35:04.503245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.503225    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nv4m"
Jul 06 20:35:04.503622 ip-10-0-139-250 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:35:04.503975 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.503957    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:35:04.504942 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.504928    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:35:04.506201 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.506182    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:35:04.510688 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.510669    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:35:04.510688 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.510686    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:04.510841 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.510758    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.511175 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511155    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:35:04.511175 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511169    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:35:04.511175 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511177    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:35:04.511378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511322    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:35:04.511378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511332    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:35:04.511679 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.510653    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-250.ec2.internal.18bfcda94de3bbdf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-250.ec2.internal,UID:ip-10-0-139-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-250.ec2.internal,},FirstTimestamp:2026-07-06 20:35:04.501943263 +0000 UTC m=+0.395261770,LastTimestamp:2026-07-06 20:35:04.501943263 +0000 UTC m=+0.395261770,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-250.ec2.internal,}"
Jul 06 20:35:04.511961 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511949    2569 factory.go:55] Registering systemd factory
Jul 06 20:35:04.512012 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.511965    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:35:04.512201 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512177    2569 factory.go:153] Registering CRI-O factory
Jul 06 20:35:04.512201 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512189    2569 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:35:04.512301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512241    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:35:04.512301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512273    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nv4m"
Jul 06 20:35:04.512301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512286    2569 factory.go:103] Registering Raw factory
Jul 06 20:35:04.512301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512303    2569 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:35:04.512688 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.512676    2569 manager.go:349] Starting recovery of all containers
Jul 06 20:35:04.522157 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.521902    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:04.526059 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.526031    2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-250.ec2.internal\" not found" node="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.526468 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.526449    2569 manager.go:354] Recovery completed
Jul 06 20:35:04.530856 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.530843    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.532843 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.532823    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.532913 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.532864    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.532913 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.532877    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.533520 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.533505    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.533612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.533506    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:35:04.533612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.533571    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:35:04.533612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.533593    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:04.534195 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534178    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.534268 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534213    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.534268 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534225    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.534733 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534720    2569 policy_none.go:49] "None policy: Start"
Jul 06 20:35:04.534847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534741    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:35:04.534847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.534756    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:35:04.535622 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.535609    2569 policy_none.go:47] "Start"
Jul 06 20:35:04.571017 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.570993    2569 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:35:04.571148 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.571036    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:35:04.571148 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571046    2569 server.go:87] "Starting device plugin registration server"
Jul 06 20:35:04.571406 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571391    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:35:04.571444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571410    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:35:04.571541 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571510    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:35:04.571638 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571619    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:35:04.571638 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.571637    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:35:04.572220 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.572198    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:35:04.572317 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.572278    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.657003 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.656964    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:35:04.658490 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.658470    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:35:04.658608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.658515    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:35:04.658608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.658543    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:35:04.658608 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.658580    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:35:04.661585 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.661564    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:04.672084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.672048    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.673074 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.673050    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.673181 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.673085    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.673181 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.673098    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.673181 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.673121    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.683218 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.683151    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.683218 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.683186    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-250.ec2.internal\": node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.706887 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.706850    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.759154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.759107    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"]
Jul 06 20:35:04.759244 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.759198    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.760789 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.760770    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.760885 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.760800    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.760885 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.760809    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.762069 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.762057    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.762203 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.762187    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.762238 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.762219    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.762988 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.762963    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.763078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.762997    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.763078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.763012    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.763078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.763019    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.763078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.763034    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.763078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.763056    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.764214 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.764198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.764319 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.764224    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:04.764888 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.764872    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:04.764949 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.764898    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:04.764949 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.764911    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:04.794017 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.793979    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-250.ec2.internal\" not found" node="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.798546 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.798527    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-250.ec2.internal\" not found" node="ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.807591 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.807569    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.812172 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.812152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.812271 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.812183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91570a53ecc3c9d70fa440a335fdf4cb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-250.ec2.internal\" (UID: \"91570a53ecc3c9d70fa440a335fdf4cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.812271 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.812207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.907881 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:04.907849    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:04.912870 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.912847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.912923 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.912881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.912923 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.912905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91570a53ecc3c9d70fa440a335fdf4cb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-250.ec2.internal\" (UID: \"91570a53ecc3c9d70fa440a335fdf4cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.912979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.912953    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.913022 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.913009    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9c22d23fde94e90c2bf4b62691b728d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal\" (UID: \"b9c22d23fde94e90c2bf4b62691b728d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:04.913057 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:04.913038    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91570a53ecc3c9d70fa440a335fdf4cb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-250.ec2.internal\" (UID: \"91570a53ecc3c9d70fa440a335fdf4cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:05.008350 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:05.008278    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:05.097810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:05.097770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:05.101628 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:05.101511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:05.109262 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:05.109234    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:05.209805 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:05.209763    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:05.310327 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:05.310245    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:05.410726 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:05.410682    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:06.611876 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.611843    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:35:06.612424 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.612029    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:35:06.612424 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.612054    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:35:06.674515 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:06.674482    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:06.676882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.676853    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:30:05 +0000 UTC" deadline="2028-04-19 05:40:11.234174176 +0000 UTC"
Jul 06 20:35:06.676882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.676882    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15657h5m4.557296363s"
Jul 06 20:35:06.775274 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:06.775197    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:06.802805 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.801325    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:06.875364 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:06.875338    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:06.882553 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:06.882524    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c22d23fde94e90c2bf4b62691b728d.slice/crio-26940c9739342dfdfa246c8181c26933032565bc402d3d02ef5e4bb682606316 WatchSource:0}: Error finding container 26940c9739342dfdfa246c8181c26933032565bc402d3d02ef5e4bb682606316: Status 404 returned error can't find the container with id 26940c9739342dfdfa246c8181c26933032565bc402d3d02ef5e4bb682606316
Jul 06 20:35:06.882827 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:06.882803    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91570a53ecc3c9d70fa440a335fdf4cb.slice/crio-b2958ff3040e96d2ef7f162cfca7cf5e4459731eec44ff0dc394e122e12103af WatchSource:0}: Error finding container b2958ff3040e96d2ef7f162cfca7cf5e4459731eec44ff0dc394e122e12103af: Status 404 returned error can't find the container with id b2958ff3040e96d2ef7f162cfca7cf5e4459731eec44ff0dc394e122e12103af
Jul 06 20:35:06.889508 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:06.889485    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:35:06.975516 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:06.975472    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:07.075964 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.075887    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:07.176737 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.176692    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-250.ec2.internal\" not found"
Jul 06 20:35:07.201658 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.201631    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:07.273897 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.273862    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:07.297753 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.297719    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:07.298780 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.298759    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal"
Jul 06 20:35:07.318429 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.318390    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:07.654666 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.654634    2569 apiserver.go:52] "Watching apiserver"
Jul 06 20:35:07.663292 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.663255    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:35:07.663792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.663768    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-xg5rc","openshift-multus/network-metrics-daemon-6wbpj","openshift-network-diagnostics/network-check-target-nwxct","openshift-network-operator/iptables-alerter-g7v8h","openshift-ovn-kubernetes/ovnkube-node-8ckcf","kube-system/global-pull-secret-syncer-flcrl","kube-system/konnectivity-agent-7bwl2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw","openshift-cluster-node-tuning-operator/tuned-kswsd","openshift-dns/node-resolver-hgq64","openshift-image-registry/node-ca-vvvlz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal","openshift-multus/multus-jp5cv"]
Jul 06 20:35:07.665247 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.665226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.666539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.666522    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.667805 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.667786    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.667893 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.667862    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:07.667975 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.667956    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7xwwc\""
Jul 06 20:35:07.668228 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.668207    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.668984 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.668958    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.669254 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.669234    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.669424 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.669406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:07.669509 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.669481    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:07.669988 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.669864    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.670360 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.670095    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tgvxv\""
Jul 06 20:35:07.670360 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.670156    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.672153 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.671593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.673463 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.673446    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:35:07.673595 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.673577    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.673797 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.673748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dw942\""
Jul 06 20:35:07.673874 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.673848    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.674070 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.674052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.674760 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.674743    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.674934 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.674918    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:07.680031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.678626    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.680031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.678794    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.680031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.679139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:35:07.680031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.679165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.681625 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.681602    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:35:07.681625 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.681616    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:35:07.682496 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682468    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.682893 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-multus\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.682996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysconfig\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.682996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.682996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-kubelet\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.682996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-env-overrides\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.682996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-var-lib-kubelet\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.682995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2063697c-84a0-4aeb-a2e5-a7b471d1590b-host-slash\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683075    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cni-binary-copy\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-etc-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-run\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-tmp\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-hostroot\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqmc\" (UniqueName: \"kubernetes.io/projected/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-kube-api-access-6wqmc\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-kubelet\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-script-lib\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-etc-kubernetes\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2063697c-84a0-4aeb-a2e5-a7b471d1590b-iptables-alerter-script\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-systemd\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-socket-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2fl\" (UniqueName: \"kubernetes.io/projected/df84e91f-55bf-4085-a78d-f5e83872d77e-kube-api-access-vx2fl\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-bin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-conf-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-systemd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-device-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c788a21-75b3-47f8-ac61-57014732d767-ovn-node-metrics-cert\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683641    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-sys\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-systemd-units\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.683765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-netns\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683714    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-ovn\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683673    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683726    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8k8gd\""
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-modprobe-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683889    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-os-release\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-socket-dir-parent\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2j7v\" (UniqueName: \"kubernetes.io/projected/f73822cb-17b8-4dde-b27f-dff1507bba12-kube-api-access-k2j7v\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-log-socket\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.683981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-config\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684025    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-conf\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9vz\" (UniqueName: \"kubernetes.io/projected/2063697c-84a0-4aeb-a2e5-a7b471d1590b-kube-api-access-gd9vz\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73822cb-17b8-4dde-b27f-dff1507bba12-tmp-dir\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-bin\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-lib-modules\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-host\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684184    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-tuned\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.684500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm2v6\" (UniqueName: \"kubernetes.io/projected/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-kube-api-access-gm2v6\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684165    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684294    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684306    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-netd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm52j\" (UniqueName: \"kubernetes.io/projected/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-kube-api-access-vm52j\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684485    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cnibin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-multus-certs\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684568    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-slash\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684641    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-kubernetes\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-daemon-config\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-sys-fs\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-k8s-cni-cncf-io\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684796    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.685124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f73822cb-17b8-4dde-b27f-dff1507bba12-hosts-file\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-registration-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-system-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-netns\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-var-lib-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-node-log\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.684984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96lh\" (UniqueName: \"kubernetes.io/projected/9c788a21-75b3-47f8-ac61-57014732d767-kube-api-access-p96lh\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.685445    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.685565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gdcx6\""
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.685680    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7p56d\""
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.685793    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ntgkv\""
Jul 06 20:35:07.686134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.685946    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:35:07.686662 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.686148    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lf2f6\""
Jul 06 20:35:07.686662 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.686410    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:35:07.689263 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.689245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.689372 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.689323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:07.690501 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.690453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.694865 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.694844    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:35:07.695100 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.695081    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-prv2s\""
Jul 06 20:35:07.695390 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.695373    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:35:07.695552 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.695531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:35:07.702148 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.702125    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:07.722078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.722050    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfhxm"
Jul 06 20:35:07.736591 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.736563    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfhxm"
Jul 06 20:35:07.775874 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.775840    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:35:07.785226 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.785391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785233    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/432ed208-35eb-45fb-801c-f1fddff9a59a-host\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.785391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-hostroot\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.785391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqmc\" (UniqueName: \"kubernetes.io/projected/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-kube-api-access-6wqmc\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.785391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-kubelet\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.785391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785371    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-hostroot\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-script-lib\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785417    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-kubelet\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-etc-kubernetes\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2063697c-84a0-4aeb-a2e5-a7b471d1590b-iptables-alerter-script\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-systemd\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-socket-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2fl\" (UniqueName: \"kubernetes.io/projected/df84e91f-55bf-4085-a78d-f5e83872d77e-kube-api-access-vx2fl\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-bin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.785637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785641    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-bin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.786114 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.785673    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-etc-kubernetes\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.786422 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786391    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-script-lib\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.786422 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786404    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2063697c-84a0-4aeb-a2e5-a7b471d1590b-iptables-alerter-script\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.786574 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-conf-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.786574 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-systemd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.786574 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786531    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-conf-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.786714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786587    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-systemd\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.786714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.786714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786663    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-socket-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.786714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cnibin\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-device-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786767    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-systemd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786795    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-device-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786800    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c788a21-75b3-47f8-ac61-57014732d767-ovn-node-metrics-cert\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-sys\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.786894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786881    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-os-release\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786957    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-sys\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-dbus\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.786996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-systemd-units\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-netns\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787104    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-ovn\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787145    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-systemd-units\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787154    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-netns\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787168    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-ovn\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-modprobe-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787219    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89105b5b-bd31-4936-8f37-f491da14518a-agent-certs\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-os-release\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-socket-dir-parent\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j7v\" (UniqueName: \"kubernetes.io/projected/f73822cb-17b8-4dde-b27f-dff1507bba12-kube-api-access-k2j7v\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.787553 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-log-socket\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787585    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-config\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-conf\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787634    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-modprobe-d\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9vz\" (UniqueName: \"kubernetes.io/projected/2063697c-84a0-4aeb-a2e5-a7b471d1590b-kube-api-access-gd9vz\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73822cb-17b8-4dde-b27f-dff1507bba12-tmp-dir\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-bin\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-lib-modules\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-host\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-tuned\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787823    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmvz\" (UniqueName: \"kubernetes.io/projected/432ed208-35eb-45fb-801c-f1fddff9a59a-kube-api-access-jxmvz\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2v6\" (UniqueName: \"kubernetes.io/projected/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-kube-api-access-gm2v6\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-netd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.787921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm52j\" (UniqueName: \"kubernetes.io/projected/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-kube-api-access-vm52j\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.787949    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-os-release\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788146    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-log-socket\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/432ed208-35eb-45fb-801c-f1fddff9a59a-serviceca\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cnibin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-multus-certs\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.788527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788535    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-lib-modules\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-slash\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-kubernetes\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-kubelet-config\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-daemon-config\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788690    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysctl-conf\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-sys-fs\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.788787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-k8s-cni-cncf-io\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f73822cb-17b8-4dde-b27f-dff1507bba12-hosts-file\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-bin\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89105b5b-bd31-4936-8f37-f491da14518a-konnectivity-ca\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788873    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-kubernetes\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-registration-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-system-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788939    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cnibin\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-netns\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-var-lib-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788991    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-host\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788984    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-multus-certs\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-node-log\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96lh\" (UniqueName: \"kubernetes.io/projected/9c788a21-75b3-47f8-ac61-57014732d767-kube-api-access-p96lh\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789084    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-slash\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.789246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789114    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789133    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-node-log\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.788954    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-ovnkube-config\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789193    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f73822cb-17b8-4dde-b27f-dff1507bba12-hosts-file\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789395    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73822cb-17b8-4dde-b27f-dff1507bba12-tmp-dir\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.789425    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789479    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-multus\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789618    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-cni-netd\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789649    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-daemon-config\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789783    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-registration-dir\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789813    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-var-lib-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.789978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.789856    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-socket-dir-parent\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.790511 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.790249    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.290202821 +0000 UTC m=+3.020768288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:07.790511 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysconfig\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.790511 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.790651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.790651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-kubelet\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.790651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-env-overrides\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.790651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-var-lib-kubelet\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.790651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:07.790862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-system-cni-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.790862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790848    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-multus-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.790953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xpj\" (UniqueName: \"kubernetes.io/projected/8f689c71-25d4-4f64-ba0a-2da7907bf33b-kube-api-access-t8xpj\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.790953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2063697c-84a0-4aeb-a2e5-a7b471d1590b-host-slash\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.790953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.790953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cni-binary-copy\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-etc-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.790980    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-sysconfig\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791032    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-netns\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791076    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-run\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-tmp\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.791164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791151    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-system-cni-dir\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791213    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-etc-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791285    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-run-k8s-cni-cncf-io\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791355    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2063697c-84a0-4aeb-a2e5-a7b471d1590b-host-slash\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791398    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-run-openvswitch\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791426    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-var-lib-kubelet\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791445    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-cni-multus\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791517    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df84e91f-55bf-4085-a78d-f5e83872d77e-sys-fs\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.791811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791574    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-host-var-lib-kubelet\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.791938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791903    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c788a21-75b3-47f8-ac61-57014732d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.792059 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.791985    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-run\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.792307 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.792200    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c788a21-75b3-47f8-ac61-57014732d767-env-overrides\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.793578 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.793537    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-etc-tuned\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.793882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.793847    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-tmp\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.793882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.793866    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c788a21-75b3-47f8-ac61-57014732d767-ovn-node-metrics-cert\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.795040 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.794977    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-cni-binary-copy\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.801618 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.801593    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm52j\" (UniqueName: \"kubernetes.io/projected/fad40002-42cf-4c56-a0b3-8ccd0e6deff3-kube-api-access-vm52j\") pod \"tuned-kswsd\" (UID: \"fad40002-42cf-4c56-a0b3-8ccd0e6deff3\") " pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.801771 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.801730    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2fl\" (UniqueName: \"kubernetes.io/projected/df84e91f-55bf-4085-a78d-f5e83872d77e-kube-api-access-vx2fl\") pod \"aws-ebs-csi-driver-node-q88nw\" (UID: \"df84e91f-55bf-4085-a78d-f5e83872d77e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:07.805806 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.805784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2v6\" (UniqueName: \"kubernetes.io/projected/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-kube-api-access-gm2v6\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:07.806859 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.806838    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:07.806859 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.806862    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:07.807042 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.806876    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:07.807042 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.806934    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.306916286 +0000 UTC m=+3.037481764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:07.809591 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.809563    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96lh\" (UniqueName: \"kubernetes.io/projected/9c788a21-75b3-47f8-ac61-57014732d767-kube-api-access-p96lh\") pod \"ovnkube-node-8ckcf\" (UID: \"9c788a21-75b3-47f8-ac61-57014732d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:07.810661 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.810639    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9vz\" (UniqueName: \"kubernetes.io/projected/2063697c-84a0-4aeb-a2e5-a7b471d1590b-kube-api-access-gd9vz\") pod \"iptables-alerter-g7v8h\" (UID: \"2063697c-84a0-4aeb-a2e5-a7b471d1590b\") " pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:07.812073 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.811872    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2j7v\" (UniqueName: \"kubernetes.io/projected/f73822cb-17b8-4dde-b27f-dff1507bba12-kube-api-access-k2j7v\") pod \"node-resolver-hgq64\" (UID: \"f73822cb-17b8-4dde-b27f-dff1507bba12\") " pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:07.817048 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.817023    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqmc\" (UniqueName: \"kubernetes.io/projected/1470ecd4-28e4-45c5-82f2-78a68a38f2d5-kube-api-access-6wqmc\") pod \"multus-jp5cv\" (UID: \"1470ecd4-28e4-45c5-82f2-78a68a38f2d5\") " pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:07.826803 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.826748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal" event={"ID":"91570a53ecc3c9d70fa440a335fdf4cb","Type":"ContainerStarted","Data":"b2958ff3040e96d2ef7f162cfca7cf5e4459731eec44ff0dc394e122e12103af"}
Jul 06 20:35:07.828033 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.827973    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal" event={"ID":"b9c22d23fde94e90c2bf4b62691b728d","Type":"ContainerStarted","Data":"26940c9739342dfdfa246c8181c26933032565bc402d3d02ef5e4bb682606316"}
Jul 06 20:35:07.833020 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.832982    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:07.891753 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-system-cni-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.891753 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xpj\" (UniqueName: \"kubernetes.io/projected/8f689c71-25d4-4f64-ba0a-2da7907bf33b-kube-api-access-t8xpj\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891825    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-system-cni-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/432ed208-35eb-45fb-801c-f1fddff9a59a-host\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cnibin\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891893    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/432ed208-35eb-45fb-801c-f1fddff9a59a-host\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-os-release\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-dbus\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.891988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89105b5b-bd31-4936-8f37-f491da14518a-agent-certs\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmvz\" (UniqueName: \"kubernetes.io/projected/432ed208-35eb-45fb-801c-f1fddff9a59a-kube-api-access-jxmvz\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/432ed208-35eb-45fb-801c-f1fddff9a59a-serviceca\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-kubelet-config\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89105b5b-bd31-4936-8f37-f491da14518a-konnectivity-ca\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892796 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892773    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892928 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892778    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.892928 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.892912    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-os-release\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.893108 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893079    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-dbus\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.893246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893225    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/432ed208-35eb-45fb-801c-f1fddff9a59a-serviceca\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.893316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893257    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.893316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893257    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.893316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893291    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f689c71-25d4-4f64-ba0a-2da7907bf33b-cnibin\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.893502 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893482    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-kubelet-config\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:07.893593 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.893577    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:07.893648 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:07.893637    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.393619478 +0000 UTC m=+3.124184958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:07.893924 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.893905    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89105b5b-bd31-4936-8f37-f491da14518a-konnectivity-ca\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.896082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.896062    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89105b5b-bd31-4936-8f37-f491da14518a-agent-certs\") pod \"konnectivity-agent-7bwl2\" (UID: \"89105b5b-bd31-4936-8f37-f491da14518a\") " pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:07.907792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.907726    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xpj\" (UniqueName: \"kubernetes.io/projected/8f689c71-25d4-4f64-ba0a-2da7907bf33b-kube-api-access-t8xpj\") pod \"multus-additional-cni-plugins-xg5rc\" (UID: \"8f689c71-25d4-4f64-ba0a-2da7907bf33b\") " pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:07.914084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.914049    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmvz\" (UniqueName: \"kubernetes.io/projected/432ed208-35eb-45fb-801c-f1fddff9a59a-kube-api-access-jxmvz\") pod \"node-ca-vvvlz\" (UID: \"432ed208-35eb-45fb-801c-f1fddff9a59a\") " pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:07.981346 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.981309    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kswsd"
Jul 06 20:35:07.990980 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:07.990956    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgq64"
Jul 06 20:35:08.003824 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.003799    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7v8h"
Jul 06 20:35:08.010905 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.010881    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:08.018832 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.018811    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw"
Jul 06 20:35:08.028708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.028683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jp5cv"
Jul 06 20:35:08.038530 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.038506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xg5rc"
Jul 06 20:35:08.048434 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.048409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:08.054305 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.054275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvvlz"
Jul 06 20:35:08.189587 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.189496    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:08.294257 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.294213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:08.294423 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.294341    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:08.294423 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.294403    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:09.294383585 +0000 UTC m=+4.024949053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:08.395540 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.395492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:08.395540 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.395546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395657    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395685    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395702    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395714    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395720    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:09.395700255 +0000 UTC m=+4.126265738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:08.395782 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:08.395759    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:09.395748316 +0000 UTC m=+4.126313780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:08.739109 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.739064    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:07 +0000 UTC" deadline="2028-03-03 02:35:08.710607237 +0000 UTC"
Jul 06 20:35:08.739109 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.739098    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14525h59m59.971512606s"
Jul 06 20:35:08.811743 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:08.811620    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73822cb_17b8_4dde_b27f_dff1507bba12.slice/crio-8b1cea8bbe3ad995d7892c1b50d3b732df76995352f10d004260a269267d3fe8 WatchSource:0}: Error finding container 8b1cea8bbe3ad995d7892c1b50d3b732df76995352f10d004260a269267d3fe8: Status 404 returned error can't find the container with id 8b1cea8bbe3ad995d7892c1b50d3b732df76995352f10d004260a269267d3fe8
Jul 06 20:35:08.815799 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:08.815685    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c788a21_75b3_47f8_ac61_57014732d767.slice/crio-32a034c76ae8eefc3a1c8db4ab6d8e1603a3ffcdda1e60d9325ad32e53219eaa WatchSource:0}: Error finding container 32a034c76ae8eefc3a1c8db4ab6d8e1603a3ffcdda1e60d9325ad32e53219eaa: Status 404 returned error can't find the container with id 32a034c76ae8eefc3a1c8db4ab6d8e1603a3ffcdda1e60d9325ad32e53219eaa
Jul 06 20:35:08.817504 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:08.817435    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1470ecd4_28e4_45c5_82f2_78a68a38f2d5.slice/crio-d643472bf6f01095a38ea78ee0508adbe692d90d385ac512a8908e88f203d540 WatchSource:0}: Error finding container d643472bf6f01095a38ea78ee0508adbe692d90d385ac512a8908e88f203d540: Status 404 returned error can't find the container with id d643472bf6f01095a38ea78ee0508adbe692d90d385ac512a8908e88f203d540
Jul 06 20:35:08.823278 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:08.818886    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f689c71_25d4_4f64_ba0a_2da7907bf33b.slice/crio-a6701de0efb186c770d16d9aeb1d2cdf590ae901d1bdca857fd08d2f317a74af WatchSource:0}: Error finding container a6701de0efb186c770d16d9aeb1d2cdf590ae901d1bdca857fd08d2f317a74af: Status 404 returned error can't find the container with id a6701de0efb186c770d16d9aeb1d2cdf590ae901d1bdca857fd08d2f317a74af
Jul 06 20:35:08.823278 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:08.821026    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89105b5b_bd31_4936_8f37_f491da14518a.slice/crio-05b9c20b1c14c304829f99261b478b93b918598d97fcf2ada8e931e56e271976 WatchSource:0}: Error finding container 05b9c20b1c14c304829f99261b478b93b918598d97fcf2ada8e931e56e271976: Status 404 returned error can't find the container with id 05b9c20b1c14c304829f99261b478b93b918598d97fcf2ada8e931e56e271976
Jul 06 20:35:08.831364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.830588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7v8h" event={"ID":"2063697c-84a0-4aeb-a2e5-a7b471d1590b","Type":"ContainerStarted","Data":"cd84e20a0807ebb22010904b4b53f523276944f5c244912c1aca41121198dfb3"}
Jul 06 20:35:08.836142 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.834773    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kswsd" event={"ID":"fad40002-42cf-4c56-a0b3-8ccd0e6deff3","Type":"ContainerStarted","Data":"08767f77692dc8f651e6d913c66953532373e9dd27f1bf6e8afb9240b1def39a"}
Jul 06 20:35:08.837514 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.837467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerStarted","Data":"a6701de0efb186c770d16d9aeb1d2cdf590ae901d1bdca857fd08d2f317a74af"}
Jul 06 20:35:08.841115 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.840990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw" event={"ID":"df84e91f-55bf-4085-a78d-f5e83872d77e","Type":"ContainerStarted","Data":"0ee692a8c49296254dc0bfc658c94c2ceb246bbdf52ee8e6edf52da9e98adc33"}
Jul 06 20:35:08.841928 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.841903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvvlz" event={"ID":"432ed208-35eb-45fb-801c-f1fddff9a59a","Type":"ContainerStarted","Data":"4ae5e311db5112303a15d2085f8e154f677144c47628b2360633f687dec80608"}
Jul 06 20:35:08.843879 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.843852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7bwl2" event={"ID":"89105b5b-bd31-4936-8f37-f491da14518a","Type":"ContainerStarted","Data":"05b9c20b1c14c304829f99261b478b93b918598d97fcf2ada8e931e56e271976"}
Jul 06 20:35:08.845079 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.845059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jp5cv" event={"ID":"1470ecd4-28e4-45c5-82f2-78a68a38f2d5","Type":"ContainerStarted","Data":"d643472bf6f01095a38ea78ee0508adbe692d90d385ac512a8908e88f203d540"}
Jul 06 20:35:08.847915 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.847892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"32a034c76ae8eefc3a1c8db4ab6d8e1603a3ffcdda1e60d9325ad32e53219eaa"}
Jul 06 20:35:08.849881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:08.849861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgq64" event={"ID":"f73822cb-17b8-4dde-b27f-dff1507bba12","Type":"ContainerStarted","Data":"8b1cea8bbe3ad995d7892c1b50d3b732df76995352f10d004260a269267d3fe8"}
Jul 06 20:35:09.301996 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.301902    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:09.302189 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.302068    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:09.302189 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.302134    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.302114333 +0000 UTC m=+6.032679797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:09.402657 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.402616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:09.402837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.402676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:09.402837 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402794    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:09.402837 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402815    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:09.402837 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402836    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:09.403070 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402849    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:09.403070 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402865    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.402845621 +0000 UTC m=+6.133411096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:09.403070 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.402971    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.402954443 +0000 UTC m=+6.133519920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:09.740041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.739909    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:07 +0000 UTC" deadline="2027-12-05 22:31:17.620348003 +0000 UTC"
Jul 06 20:35:09.740041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.739971    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12409h56m7.880379799s"
Jul 06 20:35:09.822171 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.822135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:09.822342 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.822265    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:09.822401 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.822376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:09.822476 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.822455    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:09.822530 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.822515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:09.822617 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:09.822582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:09.865504 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.864669    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9c22d23fde94e90c2bf4b62691b728d" containerID="8124e2457c7bac26dff1cd8bc7c4c5c92e929ea32535da3e4e36d9e5ce25cbc1" exitCode=0
Jul 06 20:35:09.865504 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.864775    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal" event={"ID":"b9c22d23fde94e90c2bf4b62691b728d","Type":"ContainerDied","Data":"8124e2457c7bac26dff1cd8bc7c4c5c92e929ea32535da3e4e36d9e5ce25cbc1"}
Jul 06 20:35:09.877788 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:09.877754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal" event={"ID":"91570a53ecc3c9d70fa440a335fdf4cb","Type":"ContainerStarted","Data":"67a34dd650bf0c1be46994c71488f011e5e3f6918a37305aeaef24ce85f1df9b"}
Jul 06 20:35:10.886596 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:10.886559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal" event={"ID":"b9c22d23fde94e90c2bf4b62691b728d","Type":"ContainerStarted","Data":"ac28337b2f60dc461c8caaf20caff48c4b90fd1a999d3a351c1465019bb2b089"}
Jul 06 20:35:10.911627 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:10.911544    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-250.ec2.internal" podStartSLOduration=3.911520131 podStartE2EDuration="3.911520131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:10.910624976 +0000 UTC m=+5.641190462" watchObservedRunningTime="2026-07-06 20:35:10.911520131 +0000 UTC m=+5.642085618"
Jul 06 20:35:10.912419 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:10.911787    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-250.ec2.internal" podStartSLOduration=3.91177462 podStartE2EDuration="3.91177462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:09.929310385 +0000 UTC m=+4.659875871" watchObservedRunningTime="2026-07-06 20:35:10.91177462 +0000 UTC m=+5.642340130"
Jul 06 20:35:11.317575 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.317484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:11.317745 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.317673    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:11.317745 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.317722    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:15.317709634 +0000 UTC m=+10.048275098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.418541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.418601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418706    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418766    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418780    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:15.418759705 +0000 UTC m=+10.149325172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418784    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418799    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:11.418877 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.418847    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:15.418831544 +0000 UTC m=+10.149397012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.822709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.822847    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.823254    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.823354    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:11.823412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:11.825035 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:11.823478    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:13.826116    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:13.826218    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:13.826724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:13.826796    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:13.826835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:13.826961 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:13.826892    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:15.356124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.355390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:15.356124 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.355634    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:15.356124 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.355703    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:23.35568206 +0000 UTC m=+18.086247526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:15.456742 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.456704    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:15.456917 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.456774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:15.456978 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.456952    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:15.456978 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.456974    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:15.457116 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.456985    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:15.457116 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.456988    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:15.457209 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.457061    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:23.457042959 +0000 UTC m=+18.187608436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:15.457209 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.457167    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:23.457154731 +0000 UTC m=+18.187720194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.824731    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.824864    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.824945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.825044    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:15.825096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:15.826680 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:15.825162    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:17.824481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:17.824599    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:17.824981    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:17.825103    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:17.825152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:17.825281 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:17.825223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:19.821829 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:19.821802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:19.822302 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:19.821920    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:19.822302 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:19.821802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:19.822302 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:19.822050    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:19.822571 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:19.822544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:19.822684 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:19.822661    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:21.822027 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:21.821969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:21.822469 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:21.822047    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:21.822469 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:21.821969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:21.822469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:21.822141    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:21.822469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:21.822232    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:21.822469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:21.822316    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:23.417649 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.417602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:23.418178 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.417776    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:23.418178 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.417849    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:39.417831099 +0000 UTC m=+34.148396628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:23.518227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.518182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:23.518414 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.518240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:23.518414 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518365    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:23.518414 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518364    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:23.518414 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518381    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:23.518414 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518393    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:23.518572 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518436    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:39.518419031 +0000 UTC m=+34.248984514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:23.518572 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.518452    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:39.518445789 +0000 UTC m=+34.249011252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:23.821981 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.821880    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:23.821981 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.821894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:23.822205 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:23.821902    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:23.822205 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.822030    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:23.822205 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.822115    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:23.822348 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:23.822217    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:25.822896 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:25.822873    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:25.823362 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:25.822976    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:25.823362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:25.823071    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:25.823362 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:25.823190    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:25.823362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:25.823280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:25.823490 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:25.823440    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:27.822768 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.822245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:27.822768 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.822245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:27.823388 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:27.822802    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:27.823388 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.822293    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:27.823388 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:27.822866    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:27.823388 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:27.822933    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:27.925600 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.925483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvvlz" event={"ID":"432ed208-35eb-45fb-801c-f1fddff9a59a","Type":"ContainerStarted","Data":"8a2d2ae8954e1a4595e0270fd4ace427c187f2cffc6244e5bb2a72989394516a"}
Jul 06 20:35:27.926856 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.926831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7bwl2" event={"ID":"89105b5b-bd31-4936-8f37-f491da14518a","Type":"ContainerStarted","Data":"9de52682cf0595fdc4f5f95b02ef28a4e4c9379124035eed5e7ce645d55b7081"}
Jul 06 20:35:27.928234 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.928207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jp5cv" event={"ID":"1470ecd4-28e4-45c5-82f2-78a68a38f2d5","Type":"ContainerStarted","Data":"6d650efc23e23b9e5244ab12eebb5ef6314d0aef9a471b79c4185172e3197bd5"}
Jul 06 20:35:27.930128 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.930108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"009d683ba40eae33356a9055738d36eca9611cfdec9d2c2509be8013d2c64256"}
Jul 06 20:35:27.930222 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.930177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"670cacf43ffe899d25e5577f139dceca1527151152994ebed0abb73af83d9d1f"}
Jul 06 20:35:27.930275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.930223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"b4bd47e904253f4ccda90467bb9ea73e77a25fa0705fddce927b0f62f27eab72"}
Jul 06 20:35:27.931595 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.931573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgq64" event={"ID":"f73822cb-17b8-4dde-b27f-dff1507bba12","Type":"ContainerStarted","Data":"937765cd8f186c7ea0e2ac3b3bd554f79728e9574c8295b15920cda0c72a9213"}
Jul 06 20:35:27.933056 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.933033    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kswsd" event={"ID":"fad40002-42cf-4c56-a0b3-8ccd0e6deff3","Type":"ContainerStarted","Data":"9823739a190854798d356841cc504e4a23a83438c1d14d07468f8b8250b26649"}
Jul 06 20:35:27.934391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.934369    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="09882a84174e411c47cebf596ea310e42fa6594a98237e3a7b93d210c71cbd83" exitCode=0
Jul 06 20:35:27.934464 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.934439    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"09882a84174e411c47cebf596ea310e42fa6594a98237e3a7b93d210c71cbd83"}
Jul 06 20:35:27.935766 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.935729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw" event={"ID":"df84e91f-55bf-4085-a78d-f5e83872d77e","Type":"ContainerStarted","Data":"e2b6749281cb5719200edef912265a3757cb015bd5f8ef4308c4103031c79f92"}
Jul 06 20:35:27.987076 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:27.987033    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vvvlz" podStartSLOduration=3.686359269 podStartE2EDuration="21.986996515s" totalImagesPullingTime="18.300637246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:27.950852879 +0000 UTC m=+22.681418363" watchObservedRunningTime="2026-07-06 20:35:27.986996515 +0000 UTC m=+22.717562009"
Jul 06 20:35:28.033825 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.033778    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kswsd" podStartSLOduration=4.433024486 podStartE2EDuration="23.033742051s" totalImagesPullingTime="18.600717565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:28.032815962 +0000 UTC m=+22.763381447" watchObservedRunningTime="2026-07-06 20:35:28.033742051 +0000 UTC m=+22.764307536"
Jul 06 20:35:28.052799 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.052747    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hgq64" podStartSLOduration=4.455215836 podStartE2EDuration="23.052733498s" totalImagesPullingTime="18.597517662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:28.052382899 +0000 UTC m=+22.782948383" watchObservedRunningTime="2026-07-06 20:35:28.052733498 +0000 UTC m=+22.783298981"
Jul 06 20:35:28.076994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.076945    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7bwl2" podStartSLOduration=4.776940102 podStartE2EDuration="23.076929908s" totalImagesPullingTime="18.299989806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:28.075912144 +0000 UTC m=+22.806477628" watchObservedRunningTime="2026-07-06 20:35:28.076929908 +0000 UTC m=+22.807495427"
Jul 06 20:35:28.104825 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.104771    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jp5cv" podStartSLOduration=4.440167795 podStartE2EDuration="23.104742207s" totalImagesPullingTime="18.664574412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:28.104021617 +0000 UTC m=+22.834587093" watchObservedRunningTime="2026-07-06 20:35:28.104742207 +0000 UTC m=+22.835307691"
Jul 06 20:35:28.268994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.268916    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:28.349464 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.349432    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:28.434074 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.434039    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:28.938693 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.938656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7v8h" event={"ID":"2063697c-84a0-4aeb-a2e5-a7b471d1590b","Type":"ContainerStarted","Data":"44d736724e32a515a48c87aac9b0064f00a920cadc3dfe154ce77ac96e91e03e"}
Jul 06 20:35:28.941561 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.941536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"31f79c5b46b09fc1be35f1559ebbcf2fa907361e4c81e7964a4876b4a791508f"}
Jul 06 20:35:28.941670 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.941595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"cebd6cdae71cffe292f9322e21f794468ebf1196ed7f216ac4937dc9f82ed3cf"}
Jul 06 20:35:28.941670 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.941627    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"ec07da60797f5a4f871f3f9ad78ff71f5f4b051d86f6683f62e1dc73ddae4ab7"}
Jul 06 20:35:28.963525 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:28.963498    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:35:29.123301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.123203    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.217745 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.217710    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.314507 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.314468    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.455631 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.455548    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.552623 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.552580    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.683609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.683577    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.751779 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.751616    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:28.963514184Z","UUID":"11e1683c-12c9-43cc-9628-a60a27be37f1","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:35:29.751779 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.751750    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:29.755967 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.755940    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:35:29.756127 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.755987    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:35:29.822209 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.822049    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:29.822209 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:29.822178    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:29.822602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.822584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:29.822704 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:29.822679    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:29.822836 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.822822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:29.822910 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:29.822893    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:29.945894 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:29.945856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw" event={"ID":"df84e91f-55bf-4085-a78d-f5e83872d77e","Type":"ContainerStarted","Data":"05fff69ff9c9acd91661a6ffc3edf33d87cbbb353e8d22663685a2150c297bd9"}
Jul 06 20:35:30.001159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.001131    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:30.001724 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.001708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:30.023419 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.023362    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g7v8h" podStartSLOduration=6.438889066 podStartE2EDuration="25.023339625s" totalImagesPullingTime="18.584450559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:28.964185872 +0000 UTC m=+23.694751351" watchObservedRunningTime="2026-07-06 20:35:30.023339625 +0000 UTC m=+24.753905110"
Jul 06 20:35:30.951851 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.951817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"85e3a0d88b26058eaf8961967ccdb3d06893f5b973f0f4f73c6b2d94505ee0e5"}
Jul 06 20:35:30.954100 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.954072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw" event={"ID":"df84e91f-55bf-4085-a78d-f5e83872d77e","Type":"ContainerStarted","Data":"bdcc020bf0468630695a9ea4183f5b93615aa0be759b1de54af727bba9efb41f"}
Jul 06 20:35:30.954581 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.954464    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:30.954816 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.954796    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7bwl2"
Jul 06 20:35:30.976082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:30.976032    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q88nw" podStartSLOduration=5.19465247 podStartE2EDuration="25.975999675s" totalImagesPullingTime="20.781347205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:30.97572837 +0000 UTC m=+25.706293856" watchObservedRunningTime="2026-07-06 20:35:30.975999675 +0000 UTC m=+25.706565159"
Jul 06 20:35:31.050668 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:31.050634    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:31.822165 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:31.821940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:31.822358 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:31.821940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:31.822358 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:31.822255    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:31.822358 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:31.821955    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:31.822494 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:31.822358    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:31.822494 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:31.822390    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:32.542787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.542754    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:32.640960 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.638544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:32.961409 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.961366    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" event={"ID":"9c788a21-75b3-47f8-ac61-57014732d767","Type":"ContainerStarted","Data":"8f10596795f19c2b2d6f56fe61220615ed06f6c9b102457b2a9b7384b1eb36bf"}
Jul 06 20:35:32.961637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.961618    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:32.961704 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.961679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:32.961752 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.961725    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:32.963377 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.963352    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="4a5d152e9d6e4292df293e95f4677ee490a98d25c1f56f2e86c3f9ea0a13c7e6" exitCode=0
Jul 06 20:35:32.963495 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.963428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"4a5d152e9d6e4292df293e95f4677ee490a98d25c1f56f2e86c3f9ea0a13c7e6"}
Jul 06 20:35:32.978596 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.978419    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:32.978734 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.978679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:35:32.994292 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:32.994244    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf" podStartSLOduration=9.319717366999999 podStartE2EDuration="27.994230071s" totalImagesPullingTime="18.674512704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:32.994180835 +0000 UTC m=+27.724746321" watchObservedRunningTime="2026-07-06 20:35:32.994230071 +0000 UTC m=+27.724795556"
Jul 06 20:35:33.824159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:33.824037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:33.824969 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:33.824189    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:33.824969 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:33.824621    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:33.824969 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:33.824715    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:33.824969 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:33.824799    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:33.824969 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:33.824958    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:34.827923 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.827889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flcrl"]
Jul 06 20:35:34.828426 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.827991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:34.828426 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:34.828096    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:34.830404 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.830378    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nwxct"]
Jul 06 20:35:34.830533 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.830474    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:34.830585 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:34.830565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:34.830985 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.830964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wbpj"]
Jul 06 20:35:34.831093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.831080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:34.831223 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:34.831198    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:34.969156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.969069    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="fe713935472d6a5b5ea6017ab3d78d04ab9d44119f7acfba35af8f711153727c" exitCode=0
Jul 06 20:35:34.969316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:34.969156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"fe713935472d6a5b5ea6017ab3d78d04ab9d44119f7acfba35af8f711153727c"}
Jul 06 20:35:36.821715 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:36.821519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:36.822167 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:36.821519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:36.822167 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:36.821808    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:36.822167 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:36.821524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:36.822167 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:36.821861    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:36.822167 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:36.821927    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:36.976719 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:36.976647    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="70626c207a4c3b6c55a61d1b64090932118ebe779bced3d234ac1768afd1b10a" exitCode=0
Jul 06 20:35:36.976719 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:36.976712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"70626c207a4c3b6c55a61d1b64090932118ebe779bced3d234ac1768afd1b10a"}
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:38.822190    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:38.822313    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwxct" podUID="fec65d79-bf4e-4ae6-8ece-bafbf5d1f920"
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:38.822741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:38.822837    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wbpj" podUID="8f6da2d8-84da-4bd9-b268-4414bbdce3b1"
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:38.822899    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:38.823001 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:38.822960    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flcrl" podUID="5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24"
Jul 06 20:35:39.440739 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:39.440700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:39.440943 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.440875    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:39.440943 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.440941    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:11.440922001 +0000 UTC m=+66.171487480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:39.541267 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:39.541219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:39.541273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541383    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541409    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541429    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541441    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfxhc for pod openshift-network-diagnostics/network-check-target-nwxct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:39.541461 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541448    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret podName:5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:11.541433292 +0000 UTC m=+66.271998756 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret") pod "global-pull-secret-syncer-flcrl" (UID: "5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:39.541675 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:39.541487    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc podName:fec65d79-bf4e-4ae6-8ece-bafbf5d1f920 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:11.541472644 +0000 UTC m=+66.272038127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfxhc" (UniqueName: "kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc") pod "network-check-target-nwxct" (UID: "fec65d79-bf4e-4ae6-8ece-bafbf5d1f920") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:40.051762 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.051719    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-250.ec2.internal" event="NodeReady"
Jul 06 20:35:40.052351 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.051877    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:35:40.107379 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.107335    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2j9cl"]
Jul 06 20:35:40.134829 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.134796    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjv6r"]
Jul 06 20:35:40.135035 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.134941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.137818 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.137793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:35:40.137981 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.137871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:35:40.137981 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.137881    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ng9hs\""
Jul 06 20:35:40.150938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.150904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjv6r"]
Jul 06 20:35:40.150938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.150938    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j9cl"]
Jul 06 20:35:40.151187 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.151064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.153929 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.153907    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:35:40.154259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.154242    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fq9ht\""
Jul 06 20:35:40.154350 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.154294    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:35:40.154430 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.154409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:35:40.246879 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.246793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-tmp-dir\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.246879 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.246847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-cert\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.247125 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.246912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-config-volume\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.247125 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.246961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpmf\" (UniqueName: \"kubernetes.io/projected/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-kube-api-access-sgpmf\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.247125 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.246986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2cr\" (UniqueName: \"kubernetes.io/projected/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-kube-api-access-4x2cr\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.247125 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.247065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.261924 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.261882    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:40.348136 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.348333 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-tmp-dir\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.348333 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-cert\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.348333 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-config-volume\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.348333 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpmf\" (UniqueName: \"kubernetes.io/projected/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-kube-api-access-sgpmf\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.348333 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:40.348276    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:40.348561 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:40.348350    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:40.848328759 +0000 UTC m=+35.578894244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:40.348561 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2cr\" (UniqueName: \"kubernetes.io/projected/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-kube-api-access-4x2cr\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.348674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348568    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-tmp-dir\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.349019 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.348978    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-config-volume\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.353891 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.353868    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-cert\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.358611 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.358591    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpmf\" (UniqueName: \"kubernetes.io/projected/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-kube-api-access-sgpmf\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.358715 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.358621    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2cr\" (UniqueName: \"kubernetes.io/projected/77c9edc8-8c55-4d7e-9639-143e3e6aca7c-kube-api-access-4x2cr\") pod \"ingress-canary-hjv6r\" (UID: \"77c9edc8-8c55-4d7e-9639-143e3e6aca7c\") " pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.363369 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.363338    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:40.460403 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.460365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjv6r"
Jul 06 20:35:40.822489 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.822448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:35:40.822681 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.822500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:35:40.822681 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.822503    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:35:40.826131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.825949    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gll5g\""
Jul 06 20:35:40.826131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.825992    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:35:40.826131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.826001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9pf4w\""
Jul 06 20:35:40.826131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.826039    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:35:40.826131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.826117    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:35:40.826795 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.826775    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:35:40.852841 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:40.852814    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:40.852997 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:40.852972    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:40.853094 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:40.853074    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:41.853052963 +0000 UTC m=+36.583618438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:41.288675 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:41.288441    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjv6r"]
Jul 06 20:35:41.366910 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:41.366878    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c9edc8_8c55_4d7e_9639_143e3e6aca7c.slice/crio-5d4513206188cf269dde7f7405ed39b33a3a1092300d878110d650ac34a8fedd WatchSource:0}: Error finding container 5d4513206188cf269dde7f7405ed39b33a3a1092300d878110d650ac34a8fedd: Status 404 returned error can't find the container with id 5d4513206188cf269dde7f7405ed39b33a3a1092300d878110d650ac34a8fedd
Jul 06 20:35:41.862321 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:41.862238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:41.862480 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:41.862399    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:41.862480 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:41.862469    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:43.86245418 +0000 UTC m=+38.593019643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:41.992308 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:41.992277    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="f13b102dda394d68175b6abb3781c524a6829924150ce624e3556c2469fbab1d" exitCode=0
Jul 06 20:35:41.992483 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:41.992360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"f13b102dda394d68175b6abb3781c524a6829924150ce624e3556c2469fbab1d"}
Jul 06 20:35:41.993497 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:41.993475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjv6r" event={"ID":"77c9edc8-8c55-4d7e-9639-143e3e6aca7c","Type":"ContainerStarted","Data":"5d4513206188cf269dde7f7405ed39b33a3a1092300d878110d650ac34a8fedd"}
Jul 06 20:35:42.999157 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:42.999125    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f689c71-25d4-4f64-ba0a-2da7907bf33b" containerID="ed76a8a6980dcf369a6ea7d88d3646ded08595767b0e65a8ac9660506e1e7cb8" exitCode=0
Jul 06 20:35:42.999596 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:42.999170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerDied","Data":"ed76a8a6980dcf369a6ea7d88d3646ded08595767b0e65a8ac9660506e1e7cb8"}
Jul 06 20:35:43.877084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:43.876859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:43.877261 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:43.877044    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:43.877261 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:43.877190    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:47.877172941 +0000 UTC m=+42.607738405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:44.004721 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.004673    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" event={"ID":"8f689c71-25d4-4f64-ba0a-2da7907bf33b","Type":"ContainerStarted","Data":"fd0504c6710fd923dbe83c84fa382431084bf17169480951728b9f9e512bf51b"}
Jul 06 20:35:44.006084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.006063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjv6r" event={"ID":"77c9edc8-8c55-4d7e-9639-143e3e6aca7c","Type":"ContainerStarted","Data":"35e8207281ef29bed125b3a29d648c6bfac7bff09361f5cf5aa80ea1c7a18168"}
Jul 06 20:35:44.032679 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.032631    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xg5rc" podStartSLOduration=8.887796968 podStartE2EDuration="39.032613286s" totalImagesPullingTime="30.144816318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:44.031084214 +0000 UTC m=+38.761649698" watchObservedRunningTime="2026-07-06 20:35:44.032613286 +0000 UTC m=+38.763178775"
Jul 06 20:35:44.050650 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.050598    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjv6r" podStartSLOduration=1.9403905460000002 podStartE2EDuration="4.050582816s" totalImagesPullingTime="2.11019227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:44.049269671 +0000 UTC m=+38.779835156" watchObservedRunningTime="2026-07-06 20:35:44.050582816 +0000 UTC m=+38.781148300"
Jul 06 20:35:44.134158 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.134075    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:44.197025 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.196972    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:44.255415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.255391    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:44.959181 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:44.959141    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:45.061021 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:45.060966    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:45.162039 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:45.161998    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:47.902384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:47.902346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:47.902744 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:47.902467    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:47.902744 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:47.902521    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.902507768 +0000 UTC m=+50.633073231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:50.115803 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:50.115772    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:50.190884 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:50.190833    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:50.262415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:50.262385    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:53.557198 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.557162    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-746tn"]
Jul 06 20:35:53.584868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.584838    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-98bb9bc6f-bfglc"]
Jul 06 20:35:53.585035 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.584957    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.587435 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.587413    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 06 20:35:53.587612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.587600    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:35:53.588338 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.588320    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t8fh6\""
Jul 06 20:35:53.588475 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.588458    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:35:53.589978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.589962    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 06 20:35:53.593788 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.593770    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 06 20:35:53.606792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.606771    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-746tn"]
Jul 06 20:35:53.606875 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.606801    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-98bb9bc6f-bfglc"]
Jul 06 20:35:53.606922 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.606876    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.609977 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.609958    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 06 20:35:53.610101 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.609980    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 20:35:53.610101 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.609982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 06 20:35:53.610101 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.609991    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 06 20:35:53.610425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.610401    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 20:35:53.610518 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.610449    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wh2vd\""
Jul 06 20:35:53.610858 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.610845    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 06 20:35:53.745089 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xl5\" (UniqueName: \"kubernetes.io/projected/07b86944-4f5d-4b78-8102-834bf1989fcd-kube-api-access-78xl5\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745089 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-stats-auth\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-tmp\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-default-certificate\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-snapshots\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745304 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07b86944-4f5d-4b78-8102-834bf1989fcd-serving-cert\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.745550 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.745550 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.745355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfd5s\" (UniqueName: \"kubernetes.io/projected/ca8fbc18-473f-4e4c-bffd-a44971c092ea-kube-api-access-kfd5s\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.747025 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.746985    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:53.845999 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.845907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xl5\" (UniqueName: \"kubernetes.io/projected/07b86944-4f5d-4b78-8102-834bf1989fcd-kube-api-access-78xl5\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.845999 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.845944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.845999 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.845982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-stats-auth\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-tmp\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:53.846094    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-default-certificate\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-snapshots\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07b86944-4f5d-4b78-8102-834bf1989fcd-serving-cert\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.846353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfd5s\" (UniqueName: \"kubernetes.io/projected/ca8fbc18-473f-4e4c-bffd-a44971c092ea-kube-api-access-kfd5s\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.846705 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:53.846403    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:54.34637773 +0000 UTC m=+49.076943194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:35:53.846705 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846524    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-tmp\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846705 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:53.846589    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:54.346574345 +0000 UTC m=+49.077139808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:53.846852 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846834    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/07b86944-4f5d-4b78-8102-834bf1989fcd-snapshots\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.846929 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.846909    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.847116 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.847094    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b86944-4f5d-4b78-8102-834bf1989fcd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.850415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.850394    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07b86944-4f5d-4b78-8102-834bf1989fcd-serving-cert\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.850522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.850437    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-default-certificate\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.850522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.850438    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-stats-auth\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.851139 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.851114    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:53.856951 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.856929    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xl5\" (UniqueName: \"kubernetes.io/projected/07b86944-4f5d-4b78-8102-834bf1989fcd-kube-api-access-78xl5\") pod \"insights-operator-5c847dcfc4-746tn\" (UID: \"07b86944-4f5d-4b78-8102-834bf1989fcd\") " pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:53.857061 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.856974    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfd5s\" (UniqueName: \"kubernetes.io/projected/ca8fbc18-473f-4e4c-bffd-a44971c092ea-kube-api-access-kfd5s\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:53.894515 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:53.894478    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-746tn"
Jul 06 20:35:54.069839 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:54.069810    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-746tn"]
Jul 06 20:35:54.072494 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:35:54.072465    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b86944_4f5d_4b78_8102_834bf1989fcd.slice/crio-c2a3bcd1fd4c3c084e2d9a196a4f90db4a9c473ac613607847d2422f05a89e3a WatchSource:0}: Error finding container c2a3bcd1fd4c3c084e2d9a196a4f90db4a9c473ac613607847d2422f05a89e3a: Status 404 returned error can't find the container with id c2a3bcd1fd4c3c084e2d9a196a4f90db4a9c473ac613607847d2422f05a89e3a
Jul 06 20:35:54.350161 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:54.350126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:54.350367 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:54.350168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:54.350367 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:54.350260    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:54.350367 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:54.350320    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.350298616 +0000 UTC m=+50.080864094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:54.350367 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:54.350353    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.35034166 +0000 UTC m=+50.080907129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:35:55.032675 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:55.032642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-746tn" event={"ID":"07b86944-4f5d-4b78-8102-834bf1989fcd","Type":"ContainerStarted","Data":"c2a3bcd1fd4c3c084e2d9a196a4f90db4a9c473ac613607847d2422f05a89e3a"}
Jul 06 20:35:55.359369 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:55.359280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:55.359369 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:55.359337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:55.359579 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:55.359460    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:55.359579 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:55.359472    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:57.35945546 +0000 UTC m=+52.090020923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:55.359579 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:55.359510    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:35:57.359496579 +0000 UTC m=+52.090062042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:35:55.965415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:55.965371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:35:55.965613 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:55.965537    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:55.965613 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:55.965610    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:36:11.965592507 +0000 UTC m=+66.696157974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:57.038666 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:57.038631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-746tn" event={"ID":"07b86944-4f5d-4b78-8102-834bf1989fcd","Type":"ContainerStarted","Data":"cff54a5a57816679e410e5b5d7da1dc6abf575eaedaa317192b097b210a68c43"}
Jul 06 20:35:57.059911 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:57.059854    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-746tn" podStartSLOduration=1.437070461 podStartE2EDuration="4.059835893s" totalImagesPullingTime="2.622765432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:57.05853054 +0000 UTC m=+51.789096026" watchObservedRunningTime="2026-07-06 20:35:57.059835893 +0000 UTC m=+51.790401379"
Jul 06 20:35:57.378539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:57.378445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:57.378539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:57.378502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:35:57.378766 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:57.378631    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.378606637 +0000 UTC m=+56.109172123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:57.378766 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:57.378657    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:57.378766 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:57.378704    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.378692657 +0000 UTC m=+56.109258120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:35:58.005759 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.005725    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qrfhw"]
Jul 06 20:35:58.022801 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.022769    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.025798 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.025778    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kstsn\""
Jul 06 20:35:58.026372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.026355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:35:58.026856 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.026836    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:35:58.028247 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.028187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qrfhw"]
Jul 06 20:35:58.033931 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.033904    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:58.098278 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.098236    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:58.184672 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.184638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18e46866-7fc7-4c1b-99c0-fa620cd9093f-data-volume\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.184672 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.184674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.184919 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.184731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx5p\" (UniqueName: \"kubernetes.io/projected/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-api-access-tlx5p\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.184919 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.184790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18e46866-7fc7-4c1b-99c0-fa620cd9093f-crio-socket\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.184919 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.184819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285287 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18e46866-7fc7-4c1b-99c0-fa620cd9093f-data-volume\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx5p\" (UniqueName: \"kubernetes.io/projected/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-api-access-tlx5p\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18e46866-7fc7-4c1b-99c0-fa620cd9093f-crio-socket\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285444 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.285424    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:58.285616 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.285481    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.785466783 +0000 UTC m=+53.516032247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:58.285616 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.285520    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:35:58.285616 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.285587    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.785571362 +0000 UTC m=+53.516136847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : configmap "kube-rbac-proxy" not found
Jul 06 20:35:58.285616 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285603    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18e46866-7fc7-4c1b-99c0-fa620cd9093f-crio-socket\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.285791 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.285636    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18e46866-7fc7-4c1b-99c0-fa620cd9093f-data-volume\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.299641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.299605    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx5p\" (UniqueName: \"kubernetes.io/projected/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-api-access-tlx5p\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.788902 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.788846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.789081 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:58.788939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:58.789081 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.788996    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:58.789081 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.789067    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:35:58.789081 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.789082    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:35:59.789065216 +0000 UTC m=+54.519630680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:58.789217 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:58.789128    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:35:59.789115547 +0000 UTC m=+54.519681011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.796833 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:59.796795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:59.797270 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:59.796899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:35:59.797270 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:59.796934    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.797270 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:59.797001    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.797270 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:59.797002    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.796986881 +0000 UTC m=+56.527552344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.797270 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:35:59.797076    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.797061956 +0000 UTC m=+56.527627419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.987835 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:35:59.987805    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgq64_f73822cb-17b8-4dde-b27f-dff1507bba12/dns-node-resolver/0.log"
Jul 06 20:36:00.988066 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:00.988039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvvlz_432ed208-35eb-45fb-801c-f1fddff9a59a/node-ca/0.log"
Jul 06 20:36:01.411745 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:01.411708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:01.411745 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:01.411749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:01.411946 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.411876    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:36:01.411946 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.411876    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:09.411858005 +0000 UTC m=+64.142423487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:36:01.411946 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.411926    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:09.411914051 +0000 UTC m=+64.142479515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:36:01.588559 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:01.588529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjv6r_77c9edc8-8c55-4d7e-9639-143e3e6aca7c/serve-healthcheck-canary/0.log"
Jul 06 20:36:01.815113 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:01.815072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:01.815332 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:01.815153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:01.815332 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.815234    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.815332 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.815271    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:01.815332 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.815303    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:05.815286227 +0000 UTC m=+60.545851710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.815332 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:01.815323    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:05.815311137 +0000 UTC m=+60.545876600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:04.980219 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:04.980191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ckcf"
Jul 06 20:36:05.134144 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:05.134099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:05.209055 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:05.209001    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:05.843269 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:05.843223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:05.843469 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:05.843324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:05.843469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:05.843355    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:05.843469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:05.843413    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:05.843469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:05.843429    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:13.843407731 +0000 UTC m=+68.573973194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:05.843469 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:05.843448    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:13.843438235 +0000 UTC m=+68.574003722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:06.233438 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:06.233349    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:06.344046 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:06.343991    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:08.097186 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:08.097147    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:08.209126 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:08.209083    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:08.312744 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:08.312701    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:08.387416 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:08.387338    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:08.971677 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:08.971642    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:09.079279 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:09.079243    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:09.471028 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:09.470971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:09.471028 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:09.471031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:09.471445 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:09.471139    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.471122454 +0000 UTC m=+80.201687918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : configmap references non-existent config key: service-ca.crt
Jul 06 20:36:09.471445 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:09.471181    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:36:09.471445 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:09.471229    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs podName:ca8fbc18-473f-4e4c-bffd-a44971c092ea nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.471216947 +0000 UTC m=+80.201782410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs") pod "router-default-98bb9bc6f-bfglc" (UID: "ca8fbc18-473f-4e4c-bffd-a44971c092ea") : secret "router-metrics-certs-default" not found
Jul 06 20:36:10.648347 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:10.648306    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:10.724576 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:10.724530    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.218258 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.218218    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.311772 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.311743    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.488157 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.488048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:36:11.490912 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.490893    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:36:11.499258 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:11.499240    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:36:11.499351 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:11.499304    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs podName:8f6da2d8-84da-4bd9-b268-4414bbdce3b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:15.499283704 +0000 UTC m=+130.229849185 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs") pod "network-metrics-daemon-6wbpj" (UID: "8f6da2d8-84da-4bd9-b268-4414bbdce3b1") : secret "metrics-daemon-secret" not found
Jul 06 20:36:11.588754 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.588707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:36:11.588917 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.588766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:36:11.596154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.596125    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:36:11.596154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.596127    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:36:11.601478 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.601450    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24-original-pull-secret\") pod \"global-pull-secret-syncer-flcrl\" (UID: \"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24\") " pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:36:11.606349 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.606326    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.607283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.607269    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:36:11.611538 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.611515    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfxhc\" (UniqueName: \"kubernetes.io/projected/fec65d79-bf4e-4ae6-8ece-bafbf5d1f920-kube-api-access-jfxhc\") pod \"network-check-target-nwxct\" (UID: \"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920\") " pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:36:11.737899 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.737868    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9pf4w\""
Jul 06 20:36:11.745971 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.745916    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:36:11.745971 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.745935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flcrl"
Jul 06 20:36:11.875869 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.875837    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flcrl"]
Jul 06 20:36:11.878616 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:11.878588    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d01ff7c_1c2f_4b03_94b9_fcf6b5630b24.slice/crio-87a324f3e57ed4a324f8e648ca0dfbe412f96a6cb6e9527065e7a8ec2726652c WatchSource:0}: Error finding container 87a324f3e57ed4a324f8e648ca0dfbe412f96a6cb6e9527065e7a8ec2726652c: Status 404 returned error can't find the container with id 87a324f3e57ed4a324f8e648ca0dfbe412f96a6cb6e9527065e7a8ec2726652c
Jul 06 20:36:11.892136 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.892111    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nwxct"]
Jul 06 20:36:11.894889 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:11.894866    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec65d79_bf4e_4ae6_8ece_bafbf5d1f920.slice/crio-96de83153c8941b18b880e2f5ca254c46b1f874b9e0ad6c95605b567f015bb86 WatchSource:0}: Error finding container 96de83153c8941b18b880e2f5ca254c46b1f874b9e0ad6c95605b567f015bb86: Status 404 returned error can't find the container with id 96de83153c8941b18b880e2f5ca254c46b1f874b9e0ad6c95605b567f015bb86
Jul 06 20:36:11.991271 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:11.991235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:36:11.991440 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:11.991346    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:11.991440 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:11.991398    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls podName:2abd5cc4-49a1-4779-8895-bd4e75c7e24b nodeName:}" failed. No retries permitted until 2026-07-06 20:36:43.99138404 +0000 UTC m=+98.721949504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls") pod "dns-default-2j9cl" (UID: "2abd5cc4-49a1-4779-8895-bd4e75c7e24b") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:12.072674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:12.072643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nwxct" event={"ID":"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920","Type":"ContainerStarted","Data":"96de83153c8941b18b880e2f5ca254c46b1f874b9e0ad6c95605b567f015bb86"}
Jul 06 20:36:12.076168 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:12.076141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flcrl" event={"ID":"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24","Type":"ContainerStarted","Data":"87a324f3e57ed4a324f8e648ca0dfbe412f96a6cb6e9527065e7a8ec2726652c"}
Jul 06 20:36:13.909046 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:13.908992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:13.909514 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:13.909123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:13.909514 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:13.909274    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:13.909514 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:36:13.909343    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls podName:18e46866-7fc7-4c1b-99c0-fa620cd9093f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:29.909321931 +0000 UTC m=+84.639887410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrfhw" (UID: "18e46866-7fc7-4c1b-99c0-fa620cd9093f") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:13.910360 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:13.910333    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18e46866-7fc7-4c1b-99c0-fa620cd9093f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:14.034873 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.034834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.203612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.203510    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.339602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.339559    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.404491 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.404452    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.471723 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.471631    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.638619 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.638516    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.926877 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:14.926840    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:15.524038 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:15.523985    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.126177 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:16.126146    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.329131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:16.329093    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.724290 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:16.724194    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.934774 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:16.934727    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.090254 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.090217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nwxct" event={"ID":"fec65d79-bf4e-4ae6-8ece-bafbf5d1f920","Type":"ContainerStarted","Data":"3dbd5fe19799540b8c6e6b73965f4189923b53474b7fa074d71c638a74d5a466"}
Jul 06 20:36:17.090449 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.090283    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:36:17.091654 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.091590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flcrl" event={"ID":"5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24","Type":"ContainerStarted","Data":"592bc35c5c41e15760ac1d392c7be61a73fb7230d8bad0906c92cc461be37c52"}
Jul 06 20:36:17.111352 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.111308    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nwxct" podStartSLOduration=67.589804466 podStartE2EDuration="1m12.111292259s" totalImagesPullingTime="4.521487793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:17.10950497 +0000 UTC m=+71.840070454" watchObservedRunningTime="2026-07-06 20:36:17.111292259 +0000 UTC m=+71.841857722"
Jul 06 20:36:17.127392 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.127348    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-flcrl" podStartSLOduration=66.576676628 podStartE2EDuration="1m11.127334674s" totalImagesPullingTime="4.550658046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:17.126985594 +0000 UTC m=+71.857551078" watchObservedRunningTime="2026-07-06 20:36:17.127334674 +0000 UTC m=+71.857900157"
Jul 06 20:36:17.342071 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.341944    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.527250 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.527208    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.588619 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.588582    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.728732 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.728640    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.928663 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.928633    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.993820 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:17.993705    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.070238 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:18.070202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.319393 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:18.319350    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.727318 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:18.727236    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.785639 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:18.785611    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.939892 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:18.939855    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.527514 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:19.527478    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.129748 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:20.129717    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.238943 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:20.238905    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.338540 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:20.338504    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.735716 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:20.735663    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.932468 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:20.932422    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:21.127548 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:21.127515    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:22.143843 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:22.143801    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:22.529135 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:22.529036    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:22.734824 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:22.734783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:22.938923 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:22.938883    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:23.133364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:23.133324    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:23.727738 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:23.727707    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:24.146439 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:24.146398    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:24.529860 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:24.529767    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:24.727499 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:24.727448    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:25.506351 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.506298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:25.506351 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.506357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:25.506920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.506900    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca8fbc18-473f-4e4c-bffd-a44971c092ea-service-ca-bundle\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:25.508973 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.508940    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8fbc18-473f-4e4c-bffd-a44971c092ea-metrics-certs\") pod \"router-default-98bb9bc6f-bfglc\" (UID: \"ca8fbc18-473f-4e4c-bffd-a44971c092ea\") " pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:25.718157 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.718130    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wh2vd\""
Jul 06 20:36:25.726369 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.726345    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:25.858052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:25.858025    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-98bb9bc6f-bfglc"]
Jul 06 20:36:25.861560 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:25.861526    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8fbc18_473f_4e4c_bffd_a44971c092ea.slice/crio-ec2a17e78f9a300eb6aae8753c6d4546d5efd9dec7321d70c8a17371fa75f00a WatchSource:0}: Error finding container ec2a17e78f9a300eb6aae8753c6d4546d5efd9dec7321d70c8a17371fa75f00a: Status 404 returned error can't find the container with id ec2a17e78f9a300eb6aae8753c6d4546d5efd9dec7321d70c8a17371fa75f00a
Jul 06 20:36:26.114393 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:26.114360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-98bb9bc6f-bfglc" event={"ID":"ca8fbc18-473f-4e4c-bffd-a44971c092ea","Type":"ContainerStarted","Data":"aa4ac3657da4d5c6580cc8c77ea6d8c053a2ecda8970021a104c5d0dca302ce3"}
Jul 06 20:36:26.114599 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:26.114418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-98bb9bc6f-bfglc" event={"ID":"ca8fbc18-473f-4e4c-bffd-a44971c092ea","Type":"ContainerStarted","Data":"ec2a17e78f9a300eb6aae8753c6d4546d5efd9dec7321d70c8a17371fa75f00a"}
Jul 06 20:36:26.135726 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:26.135676    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-98bb9bc6f-bfglc" podStartSLOduration=33.135657285 podStartE2EDuration="33.135657285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:26.135428845 +0000 UTC m=+80.865994330" watchObservedRunningTime="2026-07-06 20:36:26.135657285 +0000 UTC m=+80.866222771"
Jul 06 20:36:26.727104 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:26.727061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:26.729798 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:26.729777    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:27.117272 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:27.117243    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:27.118593 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:27.118567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-98bb9bc6f-bfglc"
Jul 06 20:36:28.104881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.104850    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.310841 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.310805    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.408418 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.408329    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.480126 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.480092    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.547227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.547191    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.647512 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.647477    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.817599 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.817566    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.873517 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:28.873482    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.004482 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.004451    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.065841 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.065807    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.215884 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.215750    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.313584 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.313548    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.938830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.938794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:29.941207 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:29.941185    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18e46866-7fc7-4c1b-99c0-fa620cd9093f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrfhw\" (UID: \"18e46866-7fc7-4c1b-99c0-fa620cd9093f\") " pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:30.135768 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.135738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kstsn\""
Jul 06 20:36:30.143278 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.143253    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qrfhw"
Jul 06 20:36:30.227022 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.226955    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:30.278978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.278807    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qrfhw"]
Jul 06 20:36:30.281635 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:30.281601    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e46866_7fc7_4c1b_99c0_fa620cd9093f.slice/crio-03ff0a9b51d2134b1aa8bf92b4eff6f19a9d670ac82f7a7df4b583e86d508955 WatchSource:0}: Error finding container 03ff0a9b51d2134b1aa8bf92b4eff6f19a9d670ac82f7a7df4b583e86d508955: Status 404 returned error can't find the container with id 03ff0a9b51d2134b1aa8bf92b4eff6f19a9d670ac82f7a7df4b583e86d508955
Jul 06 20:36:30.425802 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.425729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:30.490269 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:30.490185    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:31.042781 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.041765    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"]
Jul 06 20:36:31.045068 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.045041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.048413 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.048215    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4dt7t\""
Jul 06 20:36:31.049543 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.049517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 06 20:36:31.049543 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.049537    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 06 20:36:31.056882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.056789    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"]
Jul 06 20:36:31.133627 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.133587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrfhw" event={"ID":"18e46866-7fc7-4c1b-99c0-fa620cd9093f","Type":"ContainerStarted","Data":"03ff0a9b51d2134b1aa8bf92b4eff6f19a9d670ac82f7a7df4b583e86d508955"}
Jul 06 20:36:31.134053 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.134025    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:31.148605 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.148442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.148605 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.148491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.150883 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.150849    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"]
Jul 06 20:36:31.153827 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.153794    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-8fxx5"]
Jul 06 20:36:31.153962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.153938    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:31.156962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.156925    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5jnqf\""
Jul 06 20:36:31.157229 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.157211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 06 20:36:31.159873 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.158801    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"]
Jul 06 20:36:31.159873 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.159077    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:31.162428 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.162407    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.163881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.163313    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:36:31.163881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.163353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hvcnf\""
Jul 06 20:36:31.163881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.163408    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166186    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"]
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166372    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166578    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166778    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166834    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.166970    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 20:36:31.168283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.167196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 20:36:31.172921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.169874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 20:36:31.172921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.172690    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"]
Jul 06 20:36:31.172921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.172726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-8fxx5"]
Jul 06 20:36:31.246158 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.246103    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:31.249206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249285    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7b8\" (UniqueName: \"kubernetes.io/projected/c0e161d7-4f11-469d-8d0b-cd8002482390-kube-api-access-bz7b8\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg596\" (UniqueName: \"kubernetes.io/projected/ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6-kube-api-access-wg596\") pod \"downloads-685bfb5f64-8fxx5\" (UID: \"ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6\") " pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:31.249384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e161d7-4f11-469d-8d0b-cd8002482390-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249692 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.249692 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5979f59f-d2f2-4d79-ba3f-47a3100087b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-nkv2t\" (UID: \"5979f59f-d2f2-4d79-ba3f-47a3100087b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:31.249692 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.249437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.250454 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.250046    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.255444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.255377    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ad96b72-edd0-4f3d-a5f5-50a0b499493d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6m5br\" (UID: \"1ad96b72-edd0-4f3d-a5f5-50a0b499493d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.340790 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.340562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:31.349972 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.349936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350143 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.349990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350143 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7b8\" (UniqueName: \"kubernetes.io/projected/c0e161d7-4f11-469d-8d0b-cd8002482390-kube-api-access-bz7b8\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350143 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350143 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg596\" (UniqueName: \"kubernetes.io/projected/ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6-kube-api-access-wg596\") pod \"downloads-685bfb5f64-8fxx5\" (UID: \"ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6\") " pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:31.350378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e161d7-4f11-469d-8d0b-cd8002482390-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.350378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.350202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5979f59f-d2f2-4d79-ba3f-47a3100087b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-nkv2t\" (UID: \"5979f59f-d2f2-4d79-ba3f-47a3100087b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:31.351813 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.351789    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e161d7-4f11-469d-8d0b-cd8002482390-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.355548 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.355518    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.357256 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.357237    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.358029 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.357971    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5979f59f-d2f2-4d79-ba3f-47a3100087b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-nkv2t\" (UID: \"5979f59f-d2f2-4d79-ba3f-47a3100087b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:31.358129 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.358056    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.360680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.360631    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c0e161d7-4f11-469d-8d0b-cd8002482390-ca\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.362723 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.362668    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7b8\" (UniqueName: \"kubernetes.io/projected/c0e161d7-4f11-469d-8d0b-cd8002482390-kube-api-access-bz7b8\") pod \"cluster-proxy-proxy-agent-66ff9dffb6-vxc7d\" (UID: \"c0e161d7-4f11-469d-8d0b-cd8002482390\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.363199 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.363158    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg596\" (UniqueName: \"kubernetes.io/projected/ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6-kube-api-access-wg596\") pod \"downloads-685bfb5f64-8fxx5\" (UID: \"ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6\") " pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:31.365274 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.365255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"
Jul 06 20:36:31.468353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.468317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:31.484325 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.484291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:31.498478 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.498105    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"
Jul 06 20:36:31.526792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.526727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6m5br"]
Jul 06 20:36:31.641031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.640919    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"]
Jul 06 20:36:31.644172 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:31.644136    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5979f59f_d2f2_4d79_ba3f_47a3100087b4.slice/crio-616ef03af1fcf0a31171144bf46fad20b23b45ec67e0a8d65ae0ad7bf76718e2 WatchSource:0}: Error finding container 616ef03af1fcf0a31171144bf46fad20b23b45ec67e0a8d65ae0ad7bf76718e2: Status 404 returned error can't find the container with id 616ef03af1fcf0a31171144bf46fad20b23b45ec67e0a8d65ae0ad7bf76718e2
Jul 06 20:36:31.675085 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.675056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d"]
Jul 06 20:36:31.679124 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:31.679092    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e161d7_4f11_469d_8d0b_cd8002482390.slice/crio-9c4774306ba78171291c54de47a1d1632655a247490c2c093a83fdc0dc749b62 WatchSource:0}: Error finding container 9c4774306ba78171291c54de47a1d1632655a247490c2c093a83fdc0dc749b62: Status 404 returned error can't find the container with id 9c4774306ba78171291c54de47a1d1632655a247490c2c093a83fdc0dc749b62
Jul 06 20:36:31.689746 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:31.689716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-8fxx5"]
Jul 06 20:36:31.692722 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:31.692693    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d8d3e_783e_4880_b9f9_c1c9ede9e5d6.slice/crio-6b2a6345e4402256c93425e397691bbf0711e2834d69b688ae0e621cd077a712 WatchSource:0}: Error finding container 6b2a6345e4402256c93425e397691bbf0711e2834d69b688ae0e621cd077a712: Status 404 returned error can't find the container with id 6b2a6345e4402256c93425e397691bbf0711e2834d69b688ae0e621cd077a712
Jul 06 20:36:32.138095 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.138038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d" event={"ID":"c0e161d7-4f11-469d-8d0b-cd8002482390","Type":"ContainerStarted","Data":"9c4774306ba78171291c54de47a1d1632655a247490c2c093a83fdc0dc749b62"}
Jul 06 20:36:32.139420 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.139389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t" event={"ID":"5979f59f-d2f2-4d79-ba3f-47a3100087b4","Type":"ContainerStarted","Data":"616ef03af1fcf0a31171144bf46fad20b23b45ec67e0a8d65ae0ad7bf76718e2"}
Jul 06 20:36:32.140451 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.140425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-8fxx5" event={"ID":"ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6","Type":"ContainerStarted","Data":"6b2a6345e4402256c93425e397691bbf0711e2834d69b688ae0e621cd077a712"}
Jul 06 20:36:32.141617 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.141589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br" event={"ID":"1ad96b72-edd0-4f3d-a5f5-50a0b499493d","Type":"ContainerStarted","Data":"712d094bf70c6c2363128052f3e57d221cc74752fd5bcbdc85f0a0f670149c59"}
Jul 06 20:36:32.520779 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.520701    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:32.578920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:32.578821    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:37.050556 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:37.050511    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:38.168810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.168763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrfhw" event={"ID":"18e46866-7fc7-4c1b-99c0-fa620cd9093f","Type":"ContainerStarted","Data":"5591361022fd8b6d480e8e83d225fdacedf83ff7802d5b04493168d3d05e64e5"}
Jul 06 20:36:38.170699 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.170655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br" event={"ID":"1ad96b72-edd0-4f3d-a5f5-50a0b499493d","Type":"ContainerStarted","Data":"b924577a4fa87cc45dd01336df6e4dc18478ea069f0d7c5702dbd14b51333f72"}
Jul 06 20:36:38.174568 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.174536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d" event={"ID":"c0e161d7-4f11-469d-8d0b-cd8002482390","Type":"ContainerStarted","Data":"2e449153f7fd47c945b936420903c8e15ed0822c90f7912672b06ead28130168"}
Jul 06 20:36:38.174672 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.174622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d" event={"ID":"c0e161d7-4f11-469d-8d0b-cd8002482390","Type":"ContainerStarted","Data":"926bfe22c121ad4ad62475f5330c0eed5a5bdfa34cf985ff7153b7b71d541b85"}
Jul 06 20:36:38.174672 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.174646    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d" event={"ID":"c0e161d7-4f11-469d-8d0b-cd8002482390","Type":"ContainerStarted","Data":"f3c0cecadf7d878735d4ebf3e16719ce3e8821571c98b9a63df18ac8887119d0"}
Jul 06 20:36:38.176256 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.176231    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t" event={"ID":"5979f59f-d2f2-4d79-ba3f-47a3100087b4","Type":"ContainerStarted","Data":"16b0f3cbd91c854d3a1320f7d26aa2c81797943a723db350b5931c7b10e7b0dd"}
Jul 06 20:36:38.176645 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.176627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:38.181464 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.181442    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t"
Jul 06 20:36:38.188841 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.188790    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6m5br" podStartSLOduration=1.578095101 podStartE2EDuration="7.188770636s" totalImagesPullingTime="5.610675535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:38.187522395 +0000 UTC m=+92.918087881" watchObservedRunningTime="2026-07-06 20:36:38.188770636 +0000 UTC m=+92.919336123"
Jul 06 20:36:38.230270 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.230215    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-nkv2t" podStartSLOduration=1.350923811 podStartE2EDuration="7.230194966s" totalImagesPullingTime="5.879271155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:38.205202987 +0000 UTC m=+92.935768474" watchObservedRunningTime="2026-07-06 20:36:38.230194966 +0000 UTC m=+92.960760449"
Jul 06 20:36:38.231134 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.231097    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66ff9dffb6-vxc7d" podStartSLOduration=1.389220213 podStartE2EDuration="7.231087939s" totalImagesPullingTime="5.841867726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:38.229529753 +0000 UTC m=+92.960095240" watchObservedRunningTime="2026-07-06 20:36:38.231087939 +0000 UTC m=+92.961653424"
Jul 06 20:36:38.362876 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.362837    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:38.452528 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:38.452420    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:39.182927 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:39.182821    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrfhw" event={"ID":"18e46866-7fc7-4c1b-99c0-fa620cd9093f","Type":"ContainerStarted","Data":"3f18c30834129c1f5a507d04c9b3dacbfc71e888361d9398708e0a29fbba7d73"}
Jul 06 20:36:39.443801 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:39.443704    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:39.732822 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:39.732699    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:41.192081 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:41.191966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrfhw" event={"ID":"18e46866-7fc7-4c1b-99c0-fa620cd9093f","Type":"ContainerStarted","Data":"199ae986c3de4cf8f9d6aa384d439cfd008da56e2501da02dd3b2766fee99de3"}
Jul 06 20:36:41.212837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:41.212783    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qrfhw" podStartSLOduration=34.082966062 podStartE2EDuration="44.212767526s" totalImagesPullingTime="10.129801464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:41.211853175 +0000 UTC m=+95.942418653" watchObservedRunningTime="2026-07-06 20:36:41.212767526 +0000 UTC m=+95.943333011"
Jul 06 20:36:41.899721 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:41.899685    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.556647 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.556612    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.821860 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.821754    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.893803 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.893767    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"]
Jul 06 20:36:43.898388 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.898361    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:43.902259 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.902236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 06 20:36:43.903828 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.903761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 06 20:36:43.904077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.904053    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:36:43.904204 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.904101    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nrhp6\""
Jul 06 20:36:43.904204 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.904174    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:36:43.904204 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.904173    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:36:43.904376 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.904255    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:36:43.910743 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.910720    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zdsnn"]
Jul 06 20:36:43.914137 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.914112    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:43.917084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.917023    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"]
Jul 06 20:36:43.918887 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.918717    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-km4wm\""
Jul 06 20:36:43.920106 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.919545    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:36:43.920106 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.919752    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:36:43.920106 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.919960    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:36:43.929883 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:43.929849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.016707 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.016663    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.061288 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061477 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:36:44.061477 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061477 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-textfile\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061477 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-metrics-client-ca\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061477 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-sys\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw72m\" (UniqueName: \"kubernetes.io/projected/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-kube-api-access-tw72m\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-root\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87254369-84ea-43f1-9c45-6267c77a1997-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-tls\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-wtmp\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfdg\" (UniqueName: \"kubernetes.io/projected/87254369-84ea-43f1-9c45-6267c77a1997-kube-api-access-rhfdg\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.061864    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.064065 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.064044    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abd5cc4-49a1-4779-8895-bd4e75c7e24b-metrics-tls\") pod \"dns-default-2j9cl\" (UID: \"2abd5cc4-49a1-4779-8895-bd4e75c7e24b\") " pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:36:44.115053 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.114944    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.162676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-textfile\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-metrics-client-ca\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-sys\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162814    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw72m\" (UniqueName: \"kubernetes.io/projected/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-kube-api-access-tw72m\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.162868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-root\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87254369-84ea-43f1-9c45-6267c77a1997-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-tls\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-wtmp\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.162989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfdg\" (UniqueName: \"kubernetes.io/projected/87254369-84ea-43f1-9c45-6267c77a1997-kube-api-access-rhfdg\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163120    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-textfile\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163193    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-sys\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163263    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-metrics-client-ca\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163473    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163473    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163542    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163598    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-wtmp\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163635 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163610    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-root\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.163931 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.163877    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87254369-84ea-43f1-9c45-6267c77a1997-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.167194 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.167148    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.167331 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.167285    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.167394 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.167363    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87254369-84ea-43f1-9c45-6267c77a1997-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.168572 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.168552    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-node-exporter-tls\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.175672 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.175625    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfdg\" (UniqueName: \"kubernetes.io/projected/87254369-84ea-43f1-9c45-6267c77a1997-kube-api-access-rhfdg\") pod \"kube-state-metrics-7c9dbb8b7d-xpndf\" (UID: \"87254369-84ea-43f1-9c45-6267c77a1997\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.176212 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.176163    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw72m\" (UniqueName: \"kubernetes.io/projected/ac677c71-6a1c-4e48-b82a-8d6c4dd18913-kube-api-access-tw72m\") pod \"node-exporter-zdsnn\" (UID: \"ac677c71-6a1c-4e48-b82a-8d6c4dd18913\") " pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.209629 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.209602    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"
Jul 06 20:36:44.216680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.216647    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.225469 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.225444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdsnn"
Jul 06 20:36:44.315428 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.315380    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.351223 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.351182    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ng9hs\""
Jul 06 20:36:44.358649 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.358613    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:36:44.423452 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.423416    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.525658 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.525621    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.619761 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.619728    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.845199 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.845170    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.948325 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:44.948290    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:45.303616 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:45.303577    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:45.398361 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:45.398318    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:45.765741 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:45.765656    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:45.888581 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:45.888546    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.877895 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:46.877841    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.989870 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:46.989835    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:47.070071 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:47.070030    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:47.206522 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:47.206425    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.052718 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.052682    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.097041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.096993    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nwxct"
Jul 06 20:36:48.125668 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.125627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.588295 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.588264    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.612503 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.612468    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"]
Jul 06 20:36:48.615202 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.615173    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.618135 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.617938    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 20:36:48.618135 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.617940    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g2hz5\""
Jul 06 20:36:48.627746 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.627717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"]
Jul 06 20:36:48.688816 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.688778    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.705224 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.705196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmlv\" (UniqueName: \"kubernetes.io/projected/7c0e104c-11cd-411b-9362-3358b8083b59-kube-api-access-mjmlv\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.705391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.705309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c0e104c-11cd-411b-9362-3358b8083b59-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.806169 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.806136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmlv\" (UniqueName: \"kubernetes.io/projected/7c0e104c-11cd-411b-9362-3358b8083b59-kube-api-access-mjmlv\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.806349 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.806225    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c0e104c-11cd-411b-9362-3358b8083b59-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.809288 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.809258    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c0e104c-11cd-411b-9362-3358b8083b59-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.819333 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.819300    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmlv\" (UniqueName: \"kubernetes.io/projected/7c0e104c-11cd-411b-9362-3358b8083b59-kube-api-access-mjmlv\") pod \"monitoring-plugin-d5dcff7cf-db5ks\" (UID: \"7c0e104c-11cd-411b-9362-3358b8083b59\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:48.889332 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.889235    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.927376 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:48.927340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:50.228417 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:50.228361    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:50.750842 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:50.750798    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac677c71_6a1c_4e48_b82a_8d6c4dd18913.slice/crio-2c277286f4bd9cc361a38b78c07e85adcb7f6d657ad8bf069f7385b6657735f2 WatchSource:0}: Error finding container 2c277286f4bd9cc361a38b78c07e85adcb7f6d657ad8bf069f7385b6657735f2: Status 404 returned error can't find the container with id 2c277286f4bd9cc361a38b78c07e85adcb7f6d657ad8bf069f7385b6657735f2
Jul 06 20:36:50.900798 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:50.900773    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j9cl"]
Jul 06 20:36:50.903429 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:50.903402    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abd5cc4_49a1_4779_8895_bd4e75c7e24b.slice/crio-95dff1f2f86e13356405c3639e0277c658ba664a366ccc4fac6faafa9b2c040c WatchSource:0}: Error finding container 95dff1f2f86e13356405c3639e0277c658ba664a366ccc4fac6faafa9b2c040c: Status 404 returned error can't find the container with id 95dff1f2f86e13356405c3639e0277c658ba664a366ccc4fac6faafa9b2c040c
Jul 06 20:36:50.928946 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:50.928914    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"]
Jul 06 20:36:50.932427 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:50.932400    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf"]
Jul 06 20:36:50.934185 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:50.934149    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0e104c_11cd_411b_9362_3358b8083b59.slice/crio-d5c2d6da28a021fa15d8382c25c0999821a915cd7e71cb2122b2a9be09f0b098 WatchSource:0}: Error finding container d5c2d6da28a021fa15d8382c25c0999821a915cd7e71cb2122b2a9be09f0b098: Status 404 returned error can't find the container with id d5c2d6da28a021fa15d8382c25c0999821a915cd7e71cb2122b2a9be09f0b098
Jul 06 20:36:50.935564 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:36:50.935541    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87254369_84ea_43f1_9c45_6267c77a1997.slice/crio-330a01350d041f3fd0da9b1a6373cf63293c7d1e2f836f65e20a97a498a48667 WatchSource:0}: Error finding container 330a01350d041f3fd0da9b1a6373cf63293c7d1e2f836f65e20a97a498a48667: Status 404 returned error can't find the container with id 330a01350d041f3fd0da9b1a6373cf63293c7d1e2f836f65e20a97a498a48667
Jul 06 20:36:51.230349 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.230309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf" event={"ID":"87254369-84ea-43f1-9c45-6267c77a1997","Type":"ContainerStarted","Data":"330a01350d041f3fd0da9b1a6373cf63293c7d1e2f836f65e20a97a498a48667"}
Jul 06 20:36:51.232083 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.232045    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j9cl" event={"ID":"2abd5cc4-49a1-4779-8895-bd4e75c7e24b","Type":"ContainerStarted","Data":"95dff1f2f86e13356405c3639e0277c658ba664a366ccc4fac6faafa9b2c040c"}
Jul 06 20:36:51.235683 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.235337    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-8fxx5" event={"ID":"ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6","Type":"ContainerStarted","Data":"a928a07092038a9d22c571a4cd5ca452e1d1169e230afe6fb133717be6f416e9"}
Jul 06 20:36:51.235683 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.235626    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:51.236958 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.236939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsnn" event={"ID":"ac677c71-6a1c-4e48-b82a-8d6c4dd18913","Type":"ContainerStarted","Data":"2c277286f4bd9cc361a38b78c07e85adcb7f6d657ad8bf069f7385b6657735f2"}
Jul 06 20:36:51.238651 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.238626    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks" event={"ID":"7c0e104c-11cd-411b-9362-3358b8083b59","Type":"ContainerStarted","Data":"d5c2d6da28a021fa15d8382c25c0999821a915cd7e71cb2122b2a9be09f0b098"}
Jul 06 20:36:51.250284 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.250259    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-8fxx5"
Jul 06 20:36:51.285023 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:51.284498    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-8fxx5" podStartSLOduration=1.110343032 podStartE2EDuration="20.284475s" totalImagesPullingTime="19.174131968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:51.257836325 +0000 UTC m=+105.988401812" watchObservedRunningTime="2026-07-06 20:36:51.284475 +0000 UTC m=+106.015040485"
Jul 06 20:36:53.336596 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:53.336500    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:53.442336 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:53.442277    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:53.535937 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:53.535896    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:54.292178 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:54.292139    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:54.394939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:54.394861    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:54.493108 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:54.493074    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:55.260105 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.259956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsnn" event={"ID":"ac677c71-6a1c-4e48-b82a-8d6c4dd18913","Type":"ContainerStarted","Data":"897b82daa71b2d74c86d13bb99a37ecabb56d2eb9cd4f5229abf585c7e83c346"}
Jul 06 20:36:55.268915 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.265541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks" event={"ID":"7c0e104c-11cd-411b-9362-3358b8083b59","Type":"ContainerStarted","Data":"def33cd618ced7c65ed744a0657f9cc5b70ed15bb6e927b00ed0be0ef1395358"}
Jul 06 20:36:55.268915 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.266868    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:55.272612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.272466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf" event={"ID":"87254369-84ea-43f1-9c45-6267c77a1997","Type":"ContainerStarted","Data":"5c491c58ebd2a2635685bef4a467da4d29441af13b6768b14b12231f42ca874e"}
Jul 06 20:36:55.272612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.272527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf" event={"ID":"87254369-84ea-43f1-9c45-6267c77a1997","Type":"ContainerStarted","Data":"0ded16118c64893c2d37cc97ba70d71256b5101885af241aa87b29e20050efab"}
Jul 06 20:36:55.274024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.273939    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks"
Jul 06 20:36:55.276919 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.275897    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j9cl" event={"ID":"2abd5cc4-49a1-4779-8895-bd4e75c7e24b","Type":"ContainerStarted","Data":"1b657dc231c0f11cf3c0a7c417102a2cdb3997eea60de6196701e2685490a207"}
Jul 06 20:36:55.309020 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.308585    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-db5ks" podStartSLOduration=3.361035971 podStartE2EDuration="7.308563083s" totalImagesPullingTime="3.947527112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:55.305538247 +0000 UTC m=+110.036103734" watchObservedRunningTime="2026-07-06 20:36:55.308563083 +0000 UTC m=+110.039128570"
Jul 06 20:36:55.939837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:55.939804    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:56.038941 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.038903    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:56.282705 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.282669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf" event={"ID":"87254369-84ea-43f1-9c45-6267c77a1997","Type":"ContainerStarted","Data":"72a5ea5cebf2a9847e1330a17539caa8a7d3ef4fead5aded91b96edf0988db3b"}
Jul 06 20:36:56.284877 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.284842    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j9cl" event={"ID":"2abd5cc4-49a1-4779-8895-bd4e75c7e24b","Type":"ContainerStarted","Data":"ce96ef60f391f5a0922aeb892fe9402ca2ccd98feb65d38ab53a52ccd4e323af"}
Jul 06 20:36:56.285042 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.284918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:36:56.286533 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.286505    2569 generic.go:356] "Generic (PLEG): container finished" podID="ac677c71-6a1c-4e48-b82a-8d6c4dd18913" containerID="897b82daa71b2d74c86d13bb99a37ecabb56d2eb9cd4f5229abf585c7e83c346" exitCode=0
Jul 06 20:36:56.286646 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.286576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsnn" event={"ID":"ac677c71-6a1c-4e48-b82a-8d6c4dd18913","Type":"ContainerDied","Data":"897b82daa71b2d74c86d13bb99a37ecabb56d2eb9cd4f5229abf585c7e83c346"}
Jul 06 20:36:56.306268 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.306226    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xpndf" podStartSLOduration=9.359923407 podStartE2EDuration="13.306212142s" totalImagesPullingTime="3.946288735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:56.304746025 +0000 UTC m=+111.035311511" watchObservedRunningTime="2026-07-06 20:36:56.306212142 +0000 UTC m=+111.036777626"
Jul 06 20:36:56.324629 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.324562    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2j9cl" podStartSLOduration=72.341675368 podStartE2EDuration="1m16.324540967s" totalImagesPullingTime="3.982865599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:56.323652768 +0000 UTC m=+111.054218254" watchObservedRunningTime="2026-07-06 20:36:56.324540967 +0000 UTC m=+111.055106451"
Jul 06 20:36:56.496591 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.496561    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:56.594562 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.594463    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:56.696380 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:56.696342    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:57.280984 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:57.280943    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:57.297165 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:57.297129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsnn" event={"ID":"ac677c71-6a1c-4e48-b82a-8d6c4dd18913","Type":"ContainerStarted","Data":"6ef9da1280ffc27cad7b43fefe5b8404b1a370eca1ca06cc715d99b30e93bb60"}
Jul 06 20:36:57.297361 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:57.297223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsnn" event={"ID":"ac677c71-6a1c-4e48-b82a-8d6c4dd18913","Type":"ContainerStarted","Data":"70a7d7294ed56342037d9bd7c10ac75bcac7985d3f1a457a93dd98df69980604"}
Jul 06 20:36:57.334865 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:57.334807    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zdsnn" podStartSLOduration=10.034822348 podStartE2EDuration="14.334784712s" totalImagesPullingTime="4.131838935s" totalInitContainerRuntime="168.123429ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:57.333277323 +0000 UTC m=+112.063842809" watchObservedRunningTime="2026-07-06 20:36:57.334784712 +0000 UTC m=+112.065350197"
Jul 06 20:36:57.371972 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:36:57.371911    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:00.952392 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:00.952354    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.011197 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:01.011161    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.417569 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:01.417524    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.494433 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:01.494399    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:02.521046 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:02.520978    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:02.583303 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:02.583273    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:03.321193 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:03.321160    2569 generic.go:356] "Generic (PLEG): container finished" podID="07b86944-4f5d-4b78-8102-834bf1989fcd" containerID="cff54a5a57816679e410e5b5d7da1dc6abf575eaedaa317192b097b210a68c43" exitCode=0
Jul 06 20:37:03.321429 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:03.321235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-746tn" event={"ID":"07b86944-4f5d-4b78-8102-834bf1989fcd","Type":"ContainerDied","Data":"cff54a5a57816679e410e5b5d7da1dc6abf575eaedaa317192b097b210a68c43"}
Jul 06 20:37:03.321594 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:03.321580    2569 scope.go:117] "RemoveContainer" containerID="cff54a5a57816679e410e5b5d7da1dc6abf575eaedaa317192b097b210a68c43"
Jul 06 20:37:04.156708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:04.156682    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-98bb9bc6f-bfglc_ca8fbc18-473f-4e4c-bffd-a44971c092ea/router/0.log"
Jul 06 20:37:04.182237 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:04.182198    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjv6r_77c9edc8-8c55-4d7e-9639-143e3e6aca7c/serve-healthcheck-canary/0.log"
Jul 06 20:37:04.326445 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:04.326414    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-746tn" event={"ID":"07b86944-4f5d-4b78-8102-834bf1989fcd","Type":"ContainerStarted","Data":"191c5e70a5b9ff8e9fb5fc3004d071d2f0d2b87d26dea7123e2dc537bfd3af54"}
Jul 06 20:37:06.300136 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:06.300108    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2j9cl"
Jul 06 20:37:09.583734 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:09.583706    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:09.646971 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:09.646944    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:09.708706 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:09.708679    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:10.787539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:10.787492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:10.887808 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:10.887772    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:10.951725 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:10.951689    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:15.466096 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.465984    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:15.541817 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.541281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:37:15.544887 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.544824    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6da2d8-84da-4bd9-b268-4414bbdce3b1-metrics-certs\") pod \"network-metrics-daemon-6wbpj\" (UID: \"8f6da2d8-84da-4bd9-b268-4414bbdce3b1\") " pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:37:15.566220 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.566190    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:15.643800 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.643774    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gll5g\""
Jul 06 20:37:15.651370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.651342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wbpj"
Jul 06 20:37:15.851195 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:15.849180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wbpj"]
Jul 06 20:37:16.371471 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:16.371392    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wbpj" event={"ID":"8f6da2d8-84da-4bd9-b268-4414bbdce3b1","Type":"ContainerStarted","Data":"b449c8fb4526157c33d7e4867bcb2329aee9ec81e9f987cd2e6047076c3a4fd3"}
Jul 06 20:37:17.379437 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:17.379381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wbpj" event={"ID":"8f6da2d8-84da-4bd9-b268-4414bbdce3b1","Type":"ContainerStarted","Data":"9099c03c9140625717c93bb09ae58337402f2e92eb4f66ee5a4975072b286a8b"}
Jul 06 20:37:17.379863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:17.379452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wbpj" event={"ID":"8f6da2d8-84da-4bd9-b268-4414bbdce3b1","Type":"ContainerStarted","Data":"feb46c33e2c8991a08f6bc3a795cfb868e0fc74bfe5bd60a6b3f8114283167a6"}
Jul 06 20:37:17.523367 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:17.523332    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:17.597415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:37:17.597382    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:00.076334 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:00.076246    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:00.132125 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:00.132095    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:00.299627 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:00.299592    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:00.397597 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:00.397501    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:10.937231 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:10.937196    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:11.025334 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:11.025298    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:11.125130 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:11.125094    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:12.087521 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:12.087487    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:12.159722 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:12.159690    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:17.556674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:17.556634    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:17.676452 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:17.676419    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:17.768457 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:17.768423    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:34.328238 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:34.328195    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:34.390073 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:38:34.390025    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.012900 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.012815    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.118994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.118958    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.218510 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.218474    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.558275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.558241    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.656253 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.656215    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:39.736450 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:39.736415    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:40.116486 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:40.116449    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:40.187261 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:39:40.187216    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:00.018281 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:00.018241    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:00.107987 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:00.107957    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:05.689447 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:05.689416    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:40:06.490070 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:06.490031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:06.583241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:06.583204    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:12.451296 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:12.451254    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:12.709990 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:12.709899    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:12.920347 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:12.920300    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.012714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.012619    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.077244 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.077185    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6wbpj" podStartSLOduration=307.150267159 podStartE2EDuration="5m8.07716728s" totalImagesPullingTime="926.900121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:17.402329129 +0000 UTC m=+132.132894614" watchObservedRunningTime="2026-07-06 20:40:13.07716728 +0000 UTC m=+307.807732753"
Jul 06 20:40:13.078301 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.078282    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"]
Jul 06 20:40:13.080371 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.080351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.084715 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.084695    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 06 20:40:13.084935 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.084920    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 06 20:40:13.085844 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.085814    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 06 20:40:13.085844 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.085828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jb78n\""
Jul 06 20:40:13.085985 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.085855    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 06 20:40:13.085985 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.085833    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 06 20:40:13.092722 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.092702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"]
Jul 06 20:40:13.142367 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.142324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c4e59968-948a-4668-ab44-3c0b5afe3737-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.142544 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.142383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.142544 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.142432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvxr\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-kube-api-access-wkvxr\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.218137 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.218097    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.243164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.243125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.243164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.243168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvxr\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-kube-api-access-wkvxr\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.243370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.243213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c4e59968-948a-4668-ab44-3c0b5afe3737-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.243370 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:40:13.243284    2569 secret.go:281] references non-existent secret key: tls.crt
Jul 06 20:40:13.243370 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:40:13.243302    2569 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 06 20:40:13.243370 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:40:13.243318    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-qbkck: references non-existent secret key: tls.crt
Jul 06 20:40:13.243510 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:40:13.243380    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates podName:c4e59968-948a-4668-ab44-3c0b5afe3737 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:13.743365371 +0000 UTC m=+308.473930834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates") pod "keda-metrics-apiserver-559f75f947-qbkck" (UID: "c4e59968-948a-4668-ab44-3c0b5afe3737") : references non-existent secret key: tls.crt
Jul 06 20:40:13.243584 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.243568    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c4e59968-948a-4668-ab44-3c0b5afe3737-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.252680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.252656    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvxr\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-kube-api-access-wkvxr\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.712756 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.712721    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.747208 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.747175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.749756 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.749729    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c4e59968-948a-4668-ab44-3c0b5afe3737-certificates\") pod \"keda-metrics-apiserver-559f75f947-qbkck\" (UID: \"c4e59968-948a-4668-ab44-3c0b5afe3737\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:13.808812 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.808781    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.911567 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.911529    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:13.989922 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:13.989839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:14.121032 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.120953    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"]
Jul 06 20:40:14.123744 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:40:14.123717    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e59968_948a_4668_ab44_3c0b5afe3737.slice/crio-e9bfa40b34c83f37cdef2ccb8db66d41e55052f6869f2a57abee5864de02194d WatchSource:0}: Error finding container e9bfa40b34c83f37cdef2ccb8db66d41e55052f6869f2a57abee5864de02194d: Status 404 returned error can't find the container with id e9bfa40b34c83f37cdef2ccb8db66d41e55052f6869f2a57abee5864de02194d
Jul 06 20:40:14.125346 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.125328    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:40:14.210527 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.210487    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:14.395209 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.395173    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:14.495518 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.495480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:14.599450 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.599413    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:14.697510 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.697401    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:14.992343 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:14.992260    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck" event={"ID":"c4e59968-948a-4668-ab44-3c0b5afe3737","Type":"ContainerStarted","Data":"e9bfa40b34c83f37cdef2ccb8db66d41e55052f6869f2a57abee5864de02194d"}
Jul 06 20:40:18.006188 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:18.006152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck" event={"ID":"c4e59968-948a-4668-ab44-3c0b5afe3737","Type":"ContainerStarted","Data":"cf441ab84ba471a9c47c22d65280dc5ac97fcaadbeed283f803d559741e48f46"}
Jul 06 20:40:18.006603 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:18.006257    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:40:18.031189 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:18.031127    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck" podStartSLOduration=2.189831717 podStartE2EDuration="5.0311103s" totalImagesPullingTime="2.841278583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:18.029434851 +0000 UTC m=+312.760000337" watchObservedRunningTime="2026-07-06 20:40:18.0311103 +0000 UTC m=+312.761675785"
Jul 06 20:40:29.014767 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:40:29.014739    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-qbkck"
Jul 06 20:41:11.876626 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:11.876541    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:12.891432 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:12.891399    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:13.962309 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:13.962240    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:14.084674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:14.084634    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:22.868797 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.868756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:22.949097 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.949061    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:41:22.949249 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.949073    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:22.951041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.951022    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:22.954027 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.953993    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-v8wff\""
Jul 06 20:41:22.954950 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.954933    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 06 20:41:22.955052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.954957    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 20:41:22.955052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.954970    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 20:41:22.967153 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.967131    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:41:22.985783 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.985755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4xz\" (UniqueName: \"kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:22.985927 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:22.985806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.043834 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.043795    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.086440 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.086404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.086614 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.086487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4xz\" (UniqueName: \"kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.086614 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:41:23.086579    2569 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 06 20:41:23.086693 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:41:23.086677    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert podName:63e2255f-f478-4df9-9c5e-17e421b0615f nodeName:}" failed. No retries permitted until 2026-07-06 20:41:23.586654851 +0000 UTC m=+378.317220314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert") pod "kserve-controller-manager-68bc99fcb8-vfsch" (UID: "63e2255f-f478-4df9-9c5e-17e421b0615f") : secret "kserve-webhook-server-cert" not found
Jul 06 20:41:23.097277 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.097249    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4xz\" (UniqueName: \"kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.139787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.139702    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.242302 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.242269    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.300570 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.300537    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.590942 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.590908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.593330 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.593311    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") pod \"kserve-controller-manager-68bc99fcb8-vfsch\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.862046 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.861932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:23.924208 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.924164    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.992600 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.992567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:41:23.995982 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:23.995940    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:23.996142 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:41:23.996115    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e2255f_f478_4df9_9c5e_17e421b0615f.slice/crio-b40241437f1ec9c17b45068af4812871deb756faab2e246b77d5910381f2c06e WatchSource:0}: Error finding container b40241437f1ec9c17b45068af4812871deb756faab2e246b77d5910381f2c06e: Status 404 returned error can't find the container with id b40241437f1ec9c17b45068af4812871deb756faab2e246b77d5910381f2c06e
Jul 06 20:41:24.056328 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:24.056293    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:24.243346 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:24.243263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" event={"ID":"63e2255f-f478-4df9-9c5e-17e421b0615f","Type":"ContainerStarted","Data":"b40241437f1ec9c17b45068af4812871deb756faab2e246b77d5910381f2c06e"}
Jul 06 20:41:27.256420 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:27.256385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" event={"ID":"63e2255f-f478-4df9-9c5e-17e421b0615f","Type":"ContainerStarted","Data":"d9c7863c11f9b080191d6d2032c6fbfc240a723c3cd9ab5581d7e93d1a30423f"}
Jul 06 20:41:27.256830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:27.256468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:41:27.276711 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:27.276661    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" podStartSLOduration=2.879104767 podStartE2EDuration="5.276643548s" totalImagesPullingTime="2.397538781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:41:27.275591934 +0000 UTC m=+382.006157419" watchObservedRunningTime="2026-07-06 20:41:27.276643548 +0000 UTC m=+382.007209032"
Jul 06 20:41:58.265948 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:41:58.265918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:42:05.726181 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:05.726140    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:09.198252 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.198215    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:42:09.198625 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.198410    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" podUID="63e2255f-f478-4df9-9c5e-17e421b0615f" containerName="manager" containerID="cri-o://d9c7863c11f9b080191d6d2032c6fbfc240a723c3cd9ab5581d7e93d1a30423f" gracePeriod=10
Jul 06 20:42:09.230968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.230943    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-597z5"]
Jul 06 20:42:09.232834 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.232820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.243494 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.243473    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-597z5"]
Jul 06 20:42:09.345284 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.345243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51255cbe-5c11-4cd8-a95f-efef70249cec-cert\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.345462 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.345293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl22d\" (UniqueName: \"kubernetes.io/projected/51255cbe-5c11-4cd8-a95f-efef70249cec-kube-api-access-pl22d\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.408404 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.408372    2569 generic.go:356] "Generic (PLEG): container finished" podID="63e2255f-f478-4df9-9c5e-17e421b0615f" containerID="d9c7863c11f9b080191d6d2032c6fbfc240a723c3cd9ab5581d7e93d1a30423f" exitCode=0
Jul 06 20:42:09.408570 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.408425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" event={"ID":"63e2255f-f478-4df9-9c5e-17e421b0615f","Type":"ContainerDied","Data":"d9c7863c11f9b080191d6d2032c6fbfc240a723c3cd9ab5581d7e93d1a30423f"}
Jul 06 20:42:09.435572 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.435550    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:42:09.446128 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.446107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51255cbe-5c11-4cd8-a95f-efef70249cec-cert\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.446192 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.446147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl22d\" (UniqueName: \"kubernetes.io/projected/51255cbe-5c11-4cd8-a95f-efef70249cec-kube-api-access-pl22d\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.448443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.448385    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51255cbe-5c11-4cd8-a95f-efef70249cec-cert\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.456280 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.456245    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl22d\" (UniqueName: \"kubernetes.io/projected/51255cbe-5c11-4cd8-a95f-efef70249cec-kube-api-access-pl22d\") pod \"kserve-controller-manager-68bc99fcb8-597z5\" (UID: \"51255cbe-5c11-4cd8-a95f-efef70249cec\") " pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.546946 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.546912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl4xz\" (UniqueName: \"kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz\") pod \"63e2255f-f478-4df9-9c5e-17e421b0615f\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") "
Jul 06 20:42:09.547152 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.546958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") pod \"63e2255f-f478-4df9-9c5e-17e421b0615f\" (UID: \"63e2255f-f478-4df9-9c5e-17e421b0615f\") "
Jul 06 20:42:09.549274 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.549243    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz" (OuterVolumeSpecName: "kube-api-access-fl4xz") pod "63e2255f-f478-4df9-9c5e-17e421b0615f" (UID: "63e2255f-f478-4df9-9c5e-17e421b0615f"). InnerVolumeSpecName "kube-api-access-fl4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:42:09.549274 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.549262    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert" (OuterVolumeSpecName: "cert") pod "63e2255f-f478-4df9-9c5e-17e421b0615f" (UID: "63e2255f-f478-4df9-9c5e-17e421b0615f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:42:09.581592 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.581567    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:09.648374 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.648347    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4xz\" (UniqueName: \"kubernetes.io/projected/63e2255f-f478-4df9-9c5e-17e421b0615f-kube-api-access-fl4xz\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:42:09.648374 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.648373    2569 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63e2255f-f478-4df9-9c5e-17e421b0615f-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:42:09.703328 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:09.703298    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-597z5"]
Jul 06 20:42:09.705637 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:42:09.705610    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51255cbe_5c11_4cd8_a95f_efef70249cec.slice/crio-b2f323472faec5187668a4cbe39632b3b194eef238d9f6cd3d7b6af3a6e45356 WatchSource:0}: Error finding container b2f323472faec5187668a4cbe39632b3b194eef238d9f6cd3d7b6af3a6e45356: Status 404 returned error can't find the container with id b2f323472faec5187668a4cbe39632b3b194eef238d9f6cd3d7b6af3a6e45356
Jul 06 20:42:10.413123 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.412997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5" event={"ID":"51255cbe-5c11-4cd8-a95f-efef70249cec","Type":"ContainerStarted","Data":"11f46297ff155c395c162e61649c43060617f26fe4b8154ba9d091052c523586"}
Jul 06 20:42:10.413123 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.413084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5" event={"ID":"51255cbe-5c11-4cd8-a95f-efef70249cec","Type":"ContainerStarted","Data":"b2f323472faec5187668a4cbe39632b3b194eef238d9f6cd3d7b6af3a6e45356"}
Jul 06 20:42:10.413562 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.413216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:10.414311 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.414288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch" event={"ID":"63e2255f-f478-4df9-9c5e-17e421b0615f","Type":"ContainerDied","Data":"b40241437f1ec9c17b45068af4812871deb756faab2e246b77d5910381f2c06e"}
Jul 06 20:42:10.414427 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.414312    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vfsch"
Jul 06 20:42:10.414427 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.414343    2569 scope.go:117] "RemoveContainer" containerID="d9c7863c11f9b080191d6d2032c6fbfc240a723c3cd9ab5581d7e93d1a30423f"
Jul 06 20:42:10.418031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.417990    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:10.437083 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.437040    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5" podStartSLOduration=0.994432616 podStartE2EDuration="1.436999277s" totalImagesPullingTime="442.566661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:10.435571569 +0000 UTC m=+425.166137053" watchObservedRunningTime="2026-07-06 20:42:10.436999277 +0000 UTC m=+425.167564762"
Jul 06 20:42:10.454631 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.454601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:42:10.459334 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:10.459313    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vfsch"]
Jul 06 20:42:11.826866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:11.826835    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e2255f-f478-4df9-9c5e-17e421b0615f" path="/var/lib/kubelet/pods/63e2255f-f478-4df9-9c5e-17e421b0615f/volumes"
Jul 06 20:42:17.001744 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:17.001703    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:17.105860 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:17.105820    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:17.208682 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:17.208651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:41.424431 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:41.424356    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-597z5"
Jul 06 20:42:42.528978 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:42.528937    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:42.611925 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:42.611884    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:42.707441 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:42.707411    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:42.809186 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:42.809147    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:43.286740 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:43.286707    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:43.349352 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:42:43.349321    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:29.425627 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:43:29.425571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:29.498856 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:43:29.498820    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:38.157239 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:43:38.157201    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:06.366615 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:06.366502    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:18.642584 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:18.642555    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:18.743433 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:18.743397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:31.489371 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.489333    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:31.565837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.565808    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:44:31.566144 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.566130    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e2255f-f478-4df9-9c5e-17e421b0615f" containerName="manager"
Jul 06 20:44:31.566202 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.566146    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e2255f-f478-4df9-9c5e-17e421b0615f" containerName="manager"
Jul 06 20:44:31.566202 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.566197    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e2255f-f478-4df9-9c5e-17e421b0615f" containerName="manager"
Jul 06 20:44:31.569474 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.569453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.572158 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:44:31.572284 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572251    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:44:31.572513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572497    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:44:31.572599 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572528    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ea0d4-predictor-serving-cert\""
Jul 06 20:44:31.572599 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8r29g\""
Jul 06 20:44:31.572712 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572691    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:44:31.572778 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.572737    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ea0d4-kube-rbac-proxy-sar-config\""
Jul 06 20:44:31.582837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.582812    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:44:31.662232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.662197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckvl5\" (UniqueName: \"kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.662232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.662234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.662232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.662264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.662602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.662322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.662602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.662348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.664988 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.664958    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:31.762776 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.762678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.762776 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.762736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.762779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.762838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckvl5\" (UniqueName: \"kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763052 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.762865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763255 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.763229    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763519 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.763502    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.763573 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.763556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.765521 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.765498    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.772607 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.772586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckvl5\" (UniqueName: \"kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5\") pod \"isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:31.880678 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:31.880631    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:44:32.014246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:32.014221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:44:32.016694 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:44:32.016662    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb31b47_e3de_4e98_b015_7d35be23370e.slice/crio-028236cd86f6eb1f9c55a32dd0901cdbe527d7b00fb3ed17a904d777694e99cc WatchSource:0}: Error finding container 028236cd86f6eb1f9c55a32dd0901cdbe527d7b00fb3ed17a904d777694e99cc: Status 404 returned error can't find the container with id 028236cd86f6eb1f9c55a32dd0901cdbe527d7b00fb3ed17a904d777694e99cc
Jul 06 20:44:32.919337 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:32.919306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerStarted","Data":"028236cd86f6eb1f9c55a32dd0901cdbe527d7b00fb3ed17a904d777694e99cc"}
Jul 06 20:44:36.937350 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:36.937307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerStarted","Data":"a67f93d06844dba50583587ead9ef383edce3882e8a6ba7662ae6ea3d2221182"}
Jul 06 20:44:40.954213 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:40.954178    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerID="a67f93d06844dba50583587ead9ef383edce3882e8a6ba7662ae6ea3d2221182" exitCode=0
Jul 06 20:44:40.954696 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:40.954258    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerDied","Data":"a67f93d06844dba50583587ead9ef383edce3882e8a6ba7662ae6ea3d2221182"}
Jul 06 20:44:58.029410 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:44:58.029332    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerStarted","Data":"3f0047163585dc8e5ecd9e6a2fd1be8643658f42922c20b037769c3433b2bc12"}
Jul 06 20:45:01.042154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:01.042116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerStarted","Data":"362cdd061e4e78532e387d8586582ae80181ced3965d0c0c5d555fa80ea19056"}
Jul 06 20:45:04.057537 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:04.057441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerStarted","Data":"a1ee5ca3cc262ce4457a9e5f5070ff8bebc9523dc253e52cdf0f225e62054f34"}
Jul 06 20:45:04.057927 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:04.057562    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:45:04.082612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:04.082557    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podStartSLOduration=2.162130682 podStartE2EDuration="33.082541419s" totalImagesPullingTime="27.163975453s" totalInitContainerRuntime="3.756435284s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:31 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:04.081267938 +0000 UTC m=+598.811833422" watchObservedRunningTime="2026-07-06 20:45:04.082541419 +0000 UTC m=+598.813106904"
Jul 06 20:45:05.061515 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:05.061465    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:45:05.061920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:05.061551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:45:05.062917 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:05.062866    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:05.063543 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:05.063518    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:05.066393 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:05.066373    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:45:06.065643 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:06.065599    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:06.066055 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:06.065878    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:07.069138 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:07.069093    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:07.070344 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:07.069456    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:17.069681 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:17.069637    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:17.070158 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:17.070136    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:27.070115 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:27.069988    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:27.070573 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:27.070550    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:37.069508 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:37.069468    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:37.069998 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:37.069969    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:45.660369 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:45.660331    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:47.069227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:47.069185    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:47.069662 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:47.069637    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:45:57.069286 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:57.069232    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:45:57.069724 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:45:57.069699    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:46:07.069312 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:07.069262    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:46:07.069766 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:07.069680    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:46:17.069726 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:17.069694    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:17.070212 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:17.069784    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:26.767090 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.767054    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:26.851760 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.851727    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:46:26.852077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.852046    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" containerID="cri-o://3f0047163585dc8e5ecd9e6a2fd1be8643658f42922c20b037769c3433b2bc12" gracePeriod=30
Jul 06 20:46:26.852349 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.852094    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" containerID="cri-o://362cdd061e4e78532e387d8586582ae80181ced3965d0c0c5d555fa80ea19056" gracePeriod=30
Jul 06 20:46:26.852507 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.852483    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" containerID="cri-o://a1ee5ca3cc262ce4457a9e5f5070ff8bebc9523dc253e52cdf0f225e62054f34" gracePeriod=30
Jul 06 20:46:26.867714 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.867685    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:46:26.869937 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.869919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:26.873069 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.873047    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-b4160-predictor-serving-cert\""
Jul 06 20:46:26.874810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.874790    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-b4160-kube-rbac-proxy-sar-config\""
Jul 06 20:46:26.892601 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.892568    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:46:26.979244 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.979205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:26.979378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.979257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:26.979378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.979301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:26.979378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.979332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfll\" (UniqueName: \"kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:26.979478 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:26.979384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.054703 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.054674    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:46:27.057226 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.057209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.060677 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.060657    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b4160-predictor-serving-cert\""
Jul 06 20:46:27.061340 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.061321    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b4160-kube-rbac-proxy-sar-config\""
Jul 06 20:46:27.069591 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.069559    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:46:27.069863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.069843    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:46:27.077289 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.077269    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:46:27.080084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080154 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfll\" (UniqueName: \"kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080277 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080638 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080611    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080855 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.080921 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.080836    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.082771 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.082754    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.095886 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.095860    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfll\" (UniqueName: \"kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll\") pod \"isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.179789 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.179754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:27.180982 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.180959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwspj\" (UniqueName: \"kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.181079 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.180998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.181120 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.181094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.181187 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.181174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.181220 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.181198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.282566 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.282743 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwspj\" (UniqueName: \"kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.282743 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.282862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.282862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.283069 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.282987    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.283544 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.283505    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.283765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.283742    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.285446 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.285428    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.294236 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.294212    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwspj\" (UniqueName: \"kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj\") pod \"isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.317785 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.317764    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:46:27.319667 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:46:27.319642    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2375f4_5834_491d_b5c8_4cfb0f3145da.slice/crio-38f61d0534bdc48b1b9c5c43f0ef19801831631b6c4a6f71d9844a696aad7717 WatchSource:0}: Error finding container 38f61d0534bdc48b1b9c5c43f0ef19801831631b6c4a6f71d9844a696aad7717: Status 404 returned error can't find the container with id 38f61d0534bdc48b1b9c5c43f0ef19801831631b6c4a6f71d9844a696aad7717
Jul 06 20:46:27.356900 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.356827    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerID="362cdd061e4e78532e387d8586582ae80181ced3965d0c0c5d555fa80ea19056" exitCode=2
Jul 06 20:46:27.357046 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.356911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerDied","Data":"362cdd061e4e78532e387d8586582ae80181ced3965d0c0c5d555fa80ea19056"}
Jul 06 20:46:27.358162 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.358140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerStarted","Data":"38f61d0534bdc48b1b9c5c43f0ef19801831631b6c4a6f71d9844a696aad7717"}
Jul 06 20:46:27.367112 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.367092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:27.513028 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.512975    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:46:27.517455 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:46:27.517427    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ec8a99_fa22_4b93_8856_017020efa067.slice/crio-bab6251dca2c038c42aedd007e316a3ef5f0ce3e4ce87b62a80842ab8eda7ce7 WatchSource:0}: Error finding container bab6251dca2c038c42aedd007e316a3ef5f0ce3e4ce87b62a80842ab8eda7ce7: Status 404 returned error can't find the container with id bab6251dca2c038c42aedd007e316a3ef5f0ce3e4ce87b62a80842ab8eda7ce7
Jul 06 20:46:27.855236 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:27.855200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:28.363353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:28.363322    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerStarted","Data":"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"}
Jul 06 20:46:28.364869 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:28.364843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerStarted","Data":"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"}
Jul 06 20:46:28.365041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:28.364903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerStarted","Data":"bab6251dca2c038c42aedd007e316a3ef5f0ce3e4ce87b62a80842ab8eda7ce7"}
Jul 06 20:46:30.062436 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:30.062399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:31.378193 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:31.378094    2569 generic.go:356] "Generic (PLEG): container finished" podID="92ec8a99-fa22-4b93-8856-017020efa067" containerID="2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22" exitCode=0
Jul 06 20:46:31.378542 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:31.378180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerDied","Data":"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"}
Jul 06 20:46:31.379680 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:31.379664    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:46:31.379767 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:31.379713    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerID="212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb" exitCode=0
Jul 06 20:46:31.379767 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:31.379746    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerDied","Data":"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"}
Jul 06 20:46:32.387145 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.387110    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerID="3f0047163585dc8e5ecd9e6a2fd1be8643658f42922c20b037769c3433b2bc12" exitCode=0
Jul 06 20:46:32.387570 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.387181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerDied","Data":"3f0047163585dc8e5ecd9e6a2fd1be8643658f42922c20b037769c3433b2bc12"}
Jul 06 20:46:32.389765 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.389734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerStarted","Data":"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"}
Jul 06 20:46:32.389867 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.389811    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerStarted","Data":"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"}
Jul 06 20:46:32.390407 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.390137    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:32.390407 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.390387    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:32.391578 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.391551    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:46:32.414789 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:32.414736    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podStartSLOduration=2.700573966 podStartE2EDuration="6.41472141s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714147444s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:32.412602757 +0000 UTC m=+687.143168254" watchObservedRunningTime="2026-07-06 20:46:32.41472141 +0000 UTC m=+687.145286894"
Jul 06 20:46:33.395597 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:33.395521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:46:35.062515 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:35.062475    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:37.069142 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:37.069093    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:46:37.069572 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:37.069477    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:46:38.400754 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:38.400717    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:46:38.401253 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:38.401226    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:46:40.062444 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:40.062397    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:40.062842 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:40.062549    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:42.468691 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:42.468651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:42.531605 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:42.531575    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:45.062498 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:45.062450    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:47.069484 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:47.069442    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:46:47.069939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:47.069596    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:47.069939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:47.069798    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:46:47.069939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:47.069919    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:48.401529 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:48.401471    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:46:49.471207 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.471171    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerStarted","Data":"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"}
Jul 06 20:46:49.471676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.471245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerStarted","Data":"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"}
Jul 06 20:46:49.471676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.471525    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:49.471676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.471566    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:49.472821 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.472794    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:46:49.503316 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:49.503272    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podStartSLOduration=2.303697779 podStartE2EDuration="23.503259255s" totalImagesPullingTime="17.481784334s" totalInitContainerRuntime="3.717777142s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:49.502079183 +0000 UTC m=+704.232644669" watchObservedRunningTime="2026-07-06 20:46:49.503259255 +0000 UTC m=+704.233824740"
Jul 06 20:46:50.061818 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:50.061779    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:50.475454 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:50.475361    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:46:55.062256 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:55.062211    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 06 20:46:55.480563 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:55.480481    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:46:55.480956 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:55.480929    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:46:57.069463 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.069423    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent" probeResult="failure" output="Get \"http://10.134.0.20:9081/\": dial tcp 10.134.0.20:9081: connect: connection refused"
Jul 06 20:46:57.069871 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.069419    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 06 20:46:57.505423 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.505397    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerID="a1ee5ca3cc262ce4457a9e5f5070ff8bebc9523dc253e52cdf0f225e62054f34" exitCode=0
Jul 06 20:46:57.505536 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.505434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerDied","Data":"a1ee5ca3cc262ce4457a9e5f5070ff8bebc9523dc253e52cdf0f225e62054f34"}
Jul 06 20:46:57.531226 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.531149    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:57.631979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.631944    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert\") pod \"7bb31b47-e3de-4e98-b015-7d35be23370e\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") "
Jul 06 20:46:57.631979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.631986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls\") pod \"7bb31b47-e3de-4e98-b015-7d35be23370e\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") "
Jul 06 20:46:57.632245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location\") pod \"7bb31b47-e3de-4e98-b015-7d35be23370e\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") "
Jul 06 20:46:57.632245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config\") pod \"7bb31b47-e3de-4e98-b015-7d35be23370e\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") "
Jul 06 20:46:57.632245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632143    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckvl5\" (UniqueName: \"kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5\") pod \"7bb31b47-e3de-4e98-b015-7d35be23370e\" (UID: \"7bb31b47-e3de-4e98-b015-7d35be23370e\") "
Jul 06 20:46:57.632441 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632417    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bb31b47-e3de-4e98-b015-7d35be23370e" (UID: "7bb31b47-e3de-4e98-b015-7d35be23370e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:46:57.632502 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632438    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bb31b47-e3de-4e98-b015-7d35be23370e" (UID: "7bb31b47-e3de-4e98-b015-7d35be23370e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:46:57.632502 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.632465    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bb31b47-e3de-4e98-b015-7d35be23370e" (UID: "7bb31b47-e3de-4e98-b015-7d35be23370e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:46:57.634399 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.634369    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5" (OuterVolumeSpecName: "kube-api-access-ckvl5") pod "7bb31b47-e3de-4e98-b015-7d35be23370e" (UID: "7bb31b47-e3de-4e98-b015-7d35be23370e"). InnerVolumeSpecName "kube-api-access-ckvl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:46:57.634472 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.634408    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bb31b47-e3de-4e98-b015-7d35be23370e" (UID: "7bb31b47-e3de-4e98-b015-7d35be23370e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:46:57.732847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.732806    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckvl5\" (UniqueName: \"kubernetes.io/projected/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-api-access-ckvl5\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.732847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.732844    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.732847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.732855    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bb31b47-e3de-4e98-b015-7d35be23370e-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.733086 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.732863    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bb31b47-e3de-4e98-b015-7d35be23370e-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.733086 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:57.732878    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bb31b47-e3de-4e98-b015-7d35be23370e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:46:58.401749 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.401711    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:46:58.511196 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.511161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql" event={"ID":"7bb31b47-e3de-4e98-b015-7d35be23370e","Type":"ContainerDied","Data":"028236cd86f6eb1f9c55a32dd0901cdbe527d7b00fb3ed17a904d777694e99cc"}
Jul 06 20:46:58.511359 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.511231    2569 scope.go:117] "RemoveContainer" containerID="a1ee5ca3cc262ce4457a9e5f5070ff8bebc9523dc253e52cdf0f225e62054f34"
Jul 06 20:46:58.511359 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.511171    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"
Jul 06 20:46:58.518934 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.518914    2569 scope.go:117] "RemoveContainer" containerID="362cdd061e4e78532e387d8586582ae80181ced3965d0c0c5d555fa80ea19056"
Jul 06 20:46:58.526131 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.526113    2569 scope.go:117] "RemoveContainer" containerID="3f0047163585dc8e5ecd9e6a2fd1be8643658f42922c20b037769c3433b2bc12"
Jul 06 20:46:58.532907 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.532892    2569 scope.go:117] "RemoveContainer" containerID="a67f93d06844dba50583587ead9ef383edce3882e8a6ba7662ae6ea3d2221182"
Jul 06 20:46:58.540494 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.540475    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:46:58.557311 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:58.557282    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql"]
Jul 06 20:46:59.826834 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:46:59.826797    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" path="/var/lib/kubelet/pods/7bb31b47-e3de-4e98-b015-7d35be23370e/volumes"
Jul 06 20:47:05.481558 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:05.481521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:47:08.402050 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:08.401984    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:47:15.481822 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:15.481784    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:47:18.402084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:18.402045    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:47:25.481263 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:25.481221    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:47:28.402162 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:28.402121    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:47:35.481284 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:35.481245    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:47:38.401448 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:38.401411    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:47:45.481054 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:45.480991    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 06 20:47:48.401957 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:48.401930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:47:55.481754 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:47:55.481724    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:48:06.941273 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:06.941223    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:07.084503 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:07.084472    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:16.969399 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:16.969366    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.287371 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.287331    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.406815 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.406784    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:48:17.407104 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407091    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407106    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407120    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407131    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="storage-initializer"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407139    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="storage-initializer"
Jul 06 20:48:17.407156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407155    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy"
Jul 06 20:48:17.407323 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407160    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy"
Jul 06 20:48:17.407323 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407202    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kserve-container"
Jul 06 20:48:17.407323 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407209    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="agent"
Jul 06 20:48:17.407323 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.407216    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb31b47-e3de-4e98-b015-7d35be23370e" containerName="kube-rbac-proxy"
Jul 06 20:48:17.409192 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.409178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.418264 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.418243    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-88d19-kube-rbac-proxy-sar-config\""
Jul 06 20:48:17.418771 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.418748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-88d19-predictor-serving-cert\""
Jul 06 20:48:17.425208 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.425187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:48:17.472378 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.472348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.472539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.472408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.472539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.472436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.472539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.472464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgjs\" (UniqueName: \"kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.472658 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.472549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.500511 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.500474    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.516376 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.516346    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:48:17.516636 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.516610    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" containerID="cri-o://3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f" gracePeriod=30
Jul 06 20:48:17.516694 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.516637    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kube-rbac-proxy" containerID="cri-o://2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a" gracePeriod=30
Jul 06 20:48:17.573368 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgjs\" (UniqueName: \"kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.573368 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.573368 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.573618 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.573618 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.573855 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.573835    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.574040 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.574002    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.574090 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.574073    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.575930 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.575913    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.595144 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.595116    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgjs\" (UniqueName: \"kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs\") pod \"isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.602160 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.602137    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:48:17.602495 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.602475    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.604741 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.604726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.616224 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.616202    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-88d19-kube-rbac-proxy-sar-config\""
Jul 06 20:48:17.617241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.617224    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-88d19-predictor-serving-cert\""
Jul 06 20:48:17.640626 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.640601    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:48:17.674649 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.674612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.674811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.674666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kpw\" (UniqueName: \"kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.674811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.674740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.674811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.674781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.674928 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.674841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.703895 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.703865    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.718311 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.718286    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:17.767324 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.767291    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:48:17.767819 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.767785    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container" containerID="cri-o://94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04" gracePeriod=30
Jul 06 20:48:17.767987 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.767964    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kube-rbac-proxy" containerID="cri-o://d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f" gracePeriod=30
Jul 06 20:48:17.775706 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.775685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.775810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.775718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.775810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.775758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.775810 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.775783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.775951 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.775810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kpw\" (UniqueName: \"kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.776307 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.776261    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.776491 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.776465    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.776737 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.776718    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.780683 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.780657    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.803381 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.803329    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerID="2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a" exitCode=2
Jul 06 20:48:17.803516 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.803444    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerDied","Data":"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"}
Jul 06 20:48:17.804415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.804397    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kpw\" (UniqueName: \"kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw\") pod \"isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:17.823049 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.822974    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:17.875778 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.875738    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:48:17.878143 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:48:17.878118    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa27e854_5a03_43bf_b4d9_bc2f9fcf1d05.slice/crio-b3ae101a6b5472109fefcfdb2739c5f20c79247b6e5e82b7fd2969a36934ec94 WatchSource:0}: Error finding container b3ae101a6b5472109fefcfdb2739c5f20c79247b6e5e82b7fd2969a36934ec94: Status 404 returned error can't find the container with id b3ae101a6b5472109fefcfdb2739c5f20c79247b6e5e82b7fd2969a36934ec94
Jul 06 20:48:17.913885 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:17.913863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:18.052265 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.052163    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:48:18.054719 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:48:18.054694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834395f7_75f2_4a3c_a0fa_e8d3e1552e05.slice/crio-91dfec11d248a20bd99cbbe858946794558b02357206760f1ee9a5b1f0907bde WatchSource:0}: Error finding container 91dfec11d248a20bd99cbbe858946794558b02357206760f1ee9a5b1f0907bde: Status 404 returned error can't find the container with id 91dfec11d248a20bd99cbbe858946794558b02357206760f1ee9a5b1f0907bde
Jul 06 20:48:18.396307 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.396266    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 06 20:48:18.401713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.401677    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 06 20:48:18.808869 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.808841    2569 generic.go:356] "Generic (PLEG): container finished" podID="92ec8a99-fa22-4b93-8856-017020efa067" containerID="d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f" exitCode=2
Jul 06 20:48:18.809097 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.808912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerDied","Data":"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"}
Jul 06 20:48:18.810430 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.810404    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerStarted","Data":"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"}
Jul 06 20:48:18.810541 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.810461    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerStarted","Data":"b3ae101a6b5472109fefcfdb2739c5f20c79247b6e5e82b7fd2969a36934ec94"}
Jul 06 20:48:18.811796 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.811776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerStarted","Data":"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"}
Jul 06 20:48:18.811895 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:18.811816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerStarted","Data":"91dfec11d248a20bd99cbbe858946794558b02357206760f1ee9a5b1f0907bde"}
Jul 06 20:48:20.476160 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:20.476119    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 06 20:48:21.410953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.410925    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:48:21.505058 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.504933    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config\") pod \"92ec8a99-fa22-4b93-8856-017020efa067\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") "
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505091    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location\") pod \"92ec8a99-fa22-4b93-8856-017020efa067\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") "
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505130    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls\") pod \"92ec8a99-fa22-4b93-8856-017020efa067\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") "
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505166    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert\") pod \"92ec8a99-fa22-4b93-8856-017020efa067\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") "
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505202    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwspj\" (UniqueName: \"kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj\") pod \"92ec8a99-fa22-4b93-8856-017020efa067\" (UID: \"92ec8a99-fa22-4b93-8856-017020efa067\") "
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505308    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92ec8a99-fa22-4b93-8856-017020efa067" (UID: "92ec8a99-fa22-4b93-8856-017020efa067"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505368    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92ec8a99-fa22-4b93-8856-017020efa067" (UID: "92ec8a99-fa22-4b93-8856-017020efa067"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505476    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92ec8a99-fa22-4b93-8856-017020efa067-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.505513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505496    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.505779 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.505645    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92ec8a99-fa22-4b93-8856-017020efa067" (UID: "92ec8a99-fa22-4b93-8856-017020efa067"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:48:21.507405 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.507380    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92ec8a99-fa22-4b93-8856-017020efa067" (UID: "92ec8a99-fa22-4b93-8856-017020efa067"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:48:21.507514 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.507402    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj" (OuterVolumeSpecName: "kube-api-access-mwspj") pod "92ec8a99-fa22-4b93-8856-017020efa067" (UID: "92ec8a99-fa22-4b93-8856-017020efa067"). InnerVolumeSpecName "kube-api-access-mwspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:21.606245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.606190    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92ec8a99-fa22-4b93-8856-017020efa067-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.606245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.606243    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92ec8a99-fa22-4b93-8856-017020efa067-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.606245 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.606253    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwspj\" (UniqueName: \"kubernetes.io/projected/92ec8a99-fa22-4b93-8856-017020efa067-kube-api-access-mwspj\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.824813 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.824784    2569 generic.go:356] "Generic (PLEG): container finished" podID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerID="60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02" exitCode=0
Jul 06 20:48:21.826601 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.826580    2569 generic.go:356] "Generic (PLEG): container finished" podID="92ec8a99-fa22-4b93-8856-017020efa067" containerID="94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04" exitCode=0
Jul 06 20:48:21.826713 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.826695    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"
Jul 06 20:48:21.827271 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.827250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerDied","Data":"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"}
Jul 06 20:48:21.827351 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.827309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerDied","Data":"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"}
Jul 06 20:48:21.827419 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.827348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs" event={"ID":"92ec8a99-fa22-4b93-8856-017020efa067","Type":"ContainerDied","Data":"bab6251dca2c038c42aedd007e316a3ef5f0ce3e4ce87b62a80842ab8eda7ce7"}
Jul 06 20:48:21.827419 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.827403    2569 scope.go:117] "RemoveContainer" containerID="d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"
Jul 06 20:48:21.828338 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.828317    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerID="b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618" exitCode=0
Jul 06 20:48:21.828422 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.828364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerDied","Data":"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"}
Jul 06 20:48:21.845900 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.845877    2569 scope.go:117] "RemoveContainer" containerID="94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"
Jul 06 20:48:21.863249 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.863226    2569 scope.go:117] "RemoveContainer" containerID="2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"
Jul 06 20:48:21.873359 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.873342    2569 scope.go:117] "RemoveContainer" containerID="d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"
Jul 06 20:48:21.873718 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:21.873695    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f\": container with ID starting with d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f not found: ID does not exist" containerID="d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"
Jul 06 20:48:21.873807 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.873728    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f"} err="failed to get container status \"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f\": rpc error: code = NotFound desc = could not find container \"d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f\": container with ID starting with d246f08790a4497563cfee1bc32853e03945f638323c56ee9fbe7cad1fb1749f not found: ID does not exist"
Jul 06 20:48:21.873807 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.873769    2569 scope.go:117] "RemoveContainer" containerID="94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"
Jul 06 20:48:21.874068 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:21.874049    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04\": container with ID starting with 94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04 not found: ID does not exist" containerID="94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"
Jul 06 20:48:21.874140 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.874074    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04"} err="failed to get container status \"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04\": rpc error: code = NotFound desc = could not find container \"94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04\": container with ID starting with 94a302dbb0e313050fb9be264bb3951d6ec8ce65cfb3b93968ead16147206c04 not found: ID does not exist"
Jul 06 20:48:21.874140 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.874093    2569 scope.go:117] "RemoveContainer" containerID="2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"
Jul 06 20:48:21.874355 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:21.874334    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22\": container with ID starting with 2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22 not found: ID does not exist" containerID="2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"
Jul 06 20:48:21.874403 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.874363    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22"} err="failed to get container status \"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22\": rpc error: code = NotFound desc = could not find container \"2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22\": container with ID starting with 2f22b90cf270989c92983554f9b59413e9612b46e669ea537d65ce35d51b4d22 not found: ID does not exist"
Jul 06 20:48:21.899292 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.899270    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:48:21.904185 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:21.904163    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs"]
Jul 06 20:48:22.663793 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.663771    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:48:22.717569 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717542    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpfll\" (UniqueName: \"kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll\") pod \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") "
Jul 06 20:48:22.717717 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717572    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location\") pod \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") "
Jul 06 20:48:22.717717 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717616    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls\") pod \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") "
Jul 06 20:48:22.717717 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717658    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config\") pod \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") "
Jul 06 20:48:22.717717 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717698    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert\") pod \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\" (UID: \"0e2375f4-5834-491d-b5c8-4cfb0f3145da\") "
Jul 06 20:48:22.718027 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.717914    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e2375f4-5834-491d-b5c8-4cfb0f3145da" (UID: "0e2375f4-5834-491d-b5c8-4cfb0f3145da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:22.718149 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.718033    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e2375f4-5834-491d-b5c8-4cfb0f3145da" (UID: "0e2375f4-5834-491d-b5c8-4cfb0f3145da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:48:22.718149 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.718107    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e2375f4-5834-491d-b5c8-4cfb0f3145da" (UID: "0e2375f4-5834-491d-b5c8-4cfb0f3145da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:48:22.719779 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.719755    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll" (OuterVolumeSpecName: "kube-api-access-jpfll") pod "0e2375f4-5834-491d-b5c8-4cfb0f3145da" (UID: "0e2375f4-5834-491d-b5c8-4cfb0f3145da"). InnerVolumeSpecName "kube-api-access-jpfll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:22.719851 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.719830    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e2375f4-5834-491d-b5c8-4cfb0f3145da" (UID: "0e2375f4-5834-491d-b5c8-4cfb0f3145da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:48:22.818265 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.818234    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2375f4-5834-491d-b5c8-4cfb0f3145da-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:22.818265 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.818265    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:22.818452 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.818277    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e2375f4-5834-491d-b5c8-4cfb0f3145da-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:22.818452 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.818286    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpfll\" (UniqueName: \"kubernetes.io/projected/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kube-api-access-jpfll\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:22.818452 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.818296    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2375f4-5834-491d-b5c8-4cfb0f3145da-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:48:22.833873 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.833840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerStarted","Data":"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"}
Jul 06 20:48:22.834001 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.833906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerStarted","Data":"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"}
Jul 06 20:48:22.834147 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.834129    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:22.835701 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.835679    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerID="3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f" exitCode=0
Jul 06 20:48:22.835784 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.835730    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"
Jul 06 20:48:22.835784 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.835769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerDied","Data":"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"}
Jul 06 20:48:22.835854 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.835815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj" event={"ID":"0e2375f4-5834-491d-b5c8-4cfb0f3145da","Type":"ContainerDied","Data":"38f61d0534bdc48b1b9c5c43f0ef19801831631b6c4a6f71d9844a696aad7717"}
Jul 06 20:48:22.835854 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.835833    2569 scope.go:117] "RemoveContainer" containerID="2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"
Jul 06 20:48:22.838938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.838913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerStarted","Data":"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"}
Jul 06 20:48:22.839067 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.838965    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerStarted","Data":"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"}
Jul 06 20:48:22.839275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.839248    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:22.845621 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.845603    2569 scope.go:117] "RemoveContainer" containerID="3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"
Jul 06 20:48:22.853063 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.853043    2569 scope.go:117] "RemoveContainer" containerID="212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"
Jul 06 20:48:22.857827 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.857781    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podStartSLOduration=2.184069988 podStartE2EDuration="5.85776527s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.673695282s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:22.856774171 +0000 UTC m=+797.587339637" watchObservedRunningTime="2026-07-06 20:48:22.85776527 +0000 UTC m=+797.588330755"
Jul 06 20:48:22.861117 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861102    2569 scope.go:117] "RemoveContainer" containerID="2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"
Jul 06 20:48:22.861353 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:22.861333    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a\": container with ID starting with 2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a not found: ID does not exist" containerID="2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"
Jul 06 20:48:22.861440 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861359    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a"} err="failed to get container status \"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a\": rpc error: code = NotFound desc = could not find container \"2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a\": container with ID starting with 2556ea8d1f81b1961e4ef31f43d659c63961d3302804d4e324ab6b4a1f674e7a not found: ID does not exist"
Jul 06 20:48:22.861440 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861377    2569 scope.go:117] "RemoveContainer" containerID="3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"
Jul 06 20:48:22.861605 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:22.861591    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f\": container with ID starting with 3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f not found: ID does not exist" containerID="3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"
Jul 06 20:48:22.861641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861609    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f"} err="failed to get container status \"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f\": rpc error: code = NotFound desc = could not find container \"3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f\": container with ID starting with 3705a81b3af9f8a84c0f812ad0d0a1d6269330eea319a8f81b85b4a6e26b8e1f not found: ID does not exist"
Jul 06 20:48:22.861641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861622    2569 scope.go:117] "RemoveContainer" containerID="212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"
Jul 06 20:48:22.861805 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:48:22.861789    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb\": container with ID starting with 212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb not found: ID does not exist" containerID="212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"
Jul 06 20:48:22.861842 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.861811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb"} err="failed to get container status \"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb\": rpc error: code = NotFound desc = could not find container \"212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb\": container with ID starting with 212df4fe636987eace528e7ae66834acc83b03b3a2a2ac5da1ce42a2aae88bbb not found: ID does not exist"
Jul 06 20:48:22.886883 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.886831    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podStartSLOduration=2.290632064 podStartE2EDuration="5.886817136s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596185072s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:22.884798951 +0000 UTC m=+797.615364438" watchObservedRunningTime="2026-07-06 20:48:22.886817136 +0000 UTC m=+797.617382620"
Jul 06 20:48:22.902748 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.902724    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:48:22.906755 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:22.906734    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj"]
Jul 06 20:48:23.827305 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.827274    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" path="/var/lib/kubelet/pods/0e2375f4-5834-491d-b5c8-4cfb0f3145da/volumes"
Jul 06 20:48:23.827796 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.827781    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ec8a99-fa22-4b93-8856-017020efa067" path="/var/lib/kubelet/pods/92ec8a99-fa22-4b93-8856-017020efa067/volumes"
Jul 06 20:48:23.844060 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.844034    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:23.844368 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.844307    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:23.845414 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.845387    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:48:23.845516 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:23.845387    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:24.848707 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:24.848657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:48:24.849193 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:24.848793    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:29.853273 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:29.853246    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:48:29.853804 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:29.853782    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:48:29.853883 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:29.853807    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:48:29.854179 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:29.854153    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:39.853958 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:39.853921    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:48:39.854357 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:39.854168    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:49.854390 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:49.854349    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:49.854778 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:49.854353    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:48:59.854213 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:59.854177    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:48:59.854715 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:48:59.854177    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:49:09.854711 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:09.854670    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:49:09.855124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:09.854670    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:49:19.853847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:19.853812    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:49:19.854783 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:19.854760    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:49:29.854681 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:29.854641    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:49:39.854789 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:39.854758    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:49:47.674242 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:47.674204    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:57.303947 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:57.303856    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:57.792332 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:49:57.792298    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:07.404480 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.404434    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:07.689159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.689081    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:07.742811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.742782    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:50:07.743120 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743107    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kube-rbac-proxy"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743122    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kube-rbac-proxy"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743135    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="storage-initializer"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743143    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="storage-initializer"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743153    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kube-rbac-proxy"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743159    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kube-rbac-proxy"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743166    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="storage-initializer"
Jul 06 20:50:07.743170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743172    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="storage-initializer"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743183    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743188    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743197    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743256    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kube-rbac-proxy"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743266    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743271    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2375f4-5834-491d-b5c8-4cfb0f3145da" containerName="kserve-container"
Jul 06 20:50:07.743362 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.743276    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ec8a99-fa22-4b93-8856-017020efa067" containerName="kube-rbac-proxy"
Jul 06 20:50:07.747399 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.747377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.750031 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.749988    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-e7cb2-predictor-serving-cert\""
Jul 06 20:50:07.750165 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.750058    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-e7cb2-kube-rbac-proxy-sar-config\""
Jul 06 20:50:07.759289 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.759261    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:50:07.767540 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.767513    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:50:07.767825 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.767795    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" containerID="cri-o://e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f" gracePeriod=30
Jul 06 20:50:07.767952 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.767906    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kube-rbac-proxy" containerID="cri-o://196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4" gracePeriod=30
Jul 06 20:50:07.820208 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.820179    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:07.856824 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.856780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.856980 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.856871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdks\" (UniqueName: \"kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.856980 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.856970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.875601 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.875571    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:50:07.875821 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.875792    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" containerID="cri-o://0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6" gracePeriod=30
Jul 06 20:50:07.875925 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.875835    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kube-rbac-proxy" containerID="cri-o://4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a" gracePeriod=30
Jul 06 20:50:07.909876 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.909849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:07.957913 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.957838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.957913 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.957876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.958212 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.957944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdks\" (UniqueName: \"kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.958649 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.958625    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.960554 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.960531    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.968446 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.968426    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdks\" (UniqueName: \"kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks\") pod \"message-dumper-raw-e7cb2-predictor-689764df45-x5zcv\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") " pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:07.972599 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:07.972574    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:08.059721 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.059686    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:08.194597 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.194466    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:50:08.196636 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:50:08.196610    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdbbb2c_7909_4561_9234_211d7d74223b.slice/crio-89cb48dd8ee750f6b0ccd769f69c476e68413040fcae7b13bd299faee296c67d WatchSource:0}: Error finding container 89cb48dd8ee750f6b0ccd769f69c476e68413040fcae7b13bd299faee296c67d: Status 404 returned error can't find the container with id 89cb48dd8ee750f6b0ccd769f69c476e68413040fcae7b13bd299faee296c67d
Jul 06 20:50:08.233392 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.233321    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerID="196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4" exitCode=2
Jul 06 20:50:08.233504 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.233395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerDied","Data":"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"}
Jul 06 20:50:08.235348 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.235328    2569 generic.go:356] "Generic (PLEG): container finished" podID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerID="4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a" exitCode=2
Jul 06 20:50:08.235440 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.235385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerDied","Data":"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"}
Jul 06 20:50:08.236396 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:08.236380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerStarted","Data":"89cb48dd8ee750f6b0ccd769f69c476e68413040fcae7b13bd299faee296c67d"}
Jul 06 20:50:09.849543 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:09.849501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 06 20:50:09.849543 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:09.849502    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 06 20:50:09.853958 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:09.853920    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 06 20:50:09.855150 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:09.855131    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 06 20:50:10.248179 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.248071    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerStarted","Data":"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"}
Jul 06 20:50:10.248179 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.248151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerStarted","Data":"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"}
Jul 06 20:50:10.248401 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.248184    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:10.248401 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.248218    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:10.250168 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.250148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:10.273920 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:10.273871    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" podStartSLOduration=1.948394161 podStartE2EDuration="3.273855379s" totalImagesPullingTime="1.325461218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:10.269663927 +0000 UTC m=+905.000229411" watchObservedRunningTime="2026-07-06 20:50:10.273855379 +0000 UTC m=+905.004420864"
Jul 06 20:50:11.725643 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.725621    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:50:11.891364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert\") pod \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") "
Jul 06 20:50:11.891364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891315    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls\") pod \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") "
Jul 06 20:50:11.891364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891355    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config\") pod \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") "
Jul 06 20:50:11.891627 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891558    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8kpw\" (UniqueName: \"kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw\") pod \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") "
Jul 06 20:50:11.891679 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891627    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location\") pod \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\" (UID: \"834395f7-75f2-4a3c-a0fa-e8d3e1552e05\") "
Jul 06 20:50:11.891727 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891673    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "834395f7-75f2-4a3c-a0fa-e8d3e1552e05" (UID: "834395f7-75f2-4a3c-a0fa-e8d3e1552e05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:11.891727 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891694    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "834395f7-75f2-4a3c-a0fa-e8d3e1552e05" (UID: "834395f7-75f2-4a3c-a0fa-e8d3e1552e05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:11.891943 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891922    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "834395f7-75f2-4a3c-a0fa-e8d3e1552e05" (UID: "834395f7-75f2-4a3c-a0fa-e8d3e1552e05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:11.892030 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891976    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:11.892030 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.891989    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:11.893578 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.893554    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "834395f7-75f2-4a3c-a0fa-e8d3e1552e05" (UID: "834395f7-75f2-4a3c-a0fa-e8d3e1552e05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:11.893682 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.893632    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw" (OuterVolumeSpecName: "kube-api-access-c8kpw") pod "834395f7-75f2-4a3c-a0fa-e8d3e1552e05" (UID: "834395f7-75f2-4a3c-a0fa-e8d3e1552e05"). InnerVolumeSpecName "kube-api-access-c8kpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:11.993283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.993246    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kpw\" (UniqueName: \"kubernetes.io/projected/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kube-api-access-c8kpw\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:11.993283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.993274    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:11.993283 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:11.993283    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834395f7-75f2-4a3c-a0fa-e8d3e1552e05-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:12.257508 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.257408    2569 generic.go:356] "Generic (PLEG): container finished" podID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerID="0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6" exitCode=0
Jul 06 20:50:12.257508 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.257487    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"
Jul 06 20:50:12.257508 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.257497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerDied","Data":"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"}
Jul 06 20:50:12.257768 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.257554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx" event={"ID":"834395f7-75f2-4a3c-a0fa-e8d3e1552e05","Type":"ContainerDied","Data":"91dfec11d248a20bd99cbbe858946794558b02357206760f1ee9a5b1f0907bde"}
Jul 06 20:50:12.257768 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.257581    2569 scope.go:117] "RemoveContainer" containerID="4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"
Jul 06 20:50:12.266137 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.266106    2569 scope.go:117] "RemoveContainer" containerID="0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"
Jul 06 20:50:12.273590 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.273570    2569 scope.go:117] "RemoveContainer" containerID="60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"
Jul 06 20:50:12.280416 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.280400    2569 scope.go:117] "RemoveContainer" containerID="4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"
Jul 06 20:50:12.280653 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:12.280634    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a\": container with ID starting with 4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a not found: ID does not exist" containerID="4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"
Jul 06 20:50:12.280700 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.280660    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a"} err="failed to get container status \"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a\": rpc error: code = NotFound desc = could not find container \"4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a\": container with ID starting with 4f0ecc351e18b5160362a3294be41780b0b00dcb7c27e57e002e65d2efef8c6a not found: ID does not exist"
Jul 06 20:50:12.280700 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.280675    2569 scope.go:117] "RemoveContainer" containerID="0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"
Jul 06 20:50:12.280885 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:12.280871    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6\": container with ID starting with 0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6 not found: ID does not exist" containerID="0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"
Jul 06 20:50:12.280924 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.280891    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6"} err="failed to get container status \"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6\": rpc error: code = NotFound desc = could not find container \"0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6\": container with ID starting with 0a72ddac7e68326beab7ad5b1adb417502a776a22762bc86060c0a57190e74b6 not found: ID does not exist"
Jul 06 20:50:12.280924 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.280902    2569 scope.go:117] "RemoveContainer" containerID="60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"
Jul 06 20:50:12.281148 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:12.281133    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02\": container with ID starting with 60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02 not found: ID does not exist" containerID="60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"
Jul 06 20:50:12.281196 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.281152    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02"} err="failed to get container status \"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02\": rpc error: code = NotFound desc = could not find container \"60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02\": container with ID starting with 60bb5148251d0a9e194c67b07dfabe386f6096e962d2175baefc415994ee7a02 not found: ID does not exist"
Jul 06 20:50:12.287592 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.287570    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:50:12.292505 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.292485    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx"]
Jul 06 20:50:12.612820 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.612789    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:50:12.800853 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.800761    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config\") pod \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") "
Jul 06 20:50:12.800853 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.800811    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rgjs\" (UniqueName: \"kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs\") pod \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") "
Jul 06 20:50:12.801370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.800862    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls\") pod \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") "
Jul 06 20:50:12.801370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.800889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location\") pod \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") "
Jul 06 20:50:12.801370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.800937    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert\") pod \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\" (UID: \"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05\") "
Jul 06 20:50:12.801370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.801216    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" (UID: "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:12.801370 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.801309    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" (UID: "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:12.801556 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.801396    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" (UID: "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:12.803149 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.803130    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" (UID: "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:12.803250 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.803150    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs" (OuterVolumeSpecName: "kube-api-access-5rgjs") pod "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" (UID: "fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05"). InnerVolumeSpecName "kube-api-access-5rgjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:12.902132 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.902092    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:12.902132 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.902123    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:12.902132 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.902132    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgjs\" (UniqueName: \"kubernetes.io/projected/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kube-api-access-5rgjs\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:12.902132 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.902142    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:12.902391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:12.902151    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:50:13.262910 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.262820    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerID="e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f" exitCode=0
Jul 06 20:50:13.262910 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.262885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"
Jul 06 20:50:13.263156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.262906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerDied","Data":"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"}
Jul 06 20:50:13.263156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.262967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn" event={"ID":"fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05","Type":"ContainerDied","Data":"b3ae101a6b5472109fefcfdb2739c5f20c79247b6e5e82b7fd2969a36934ec94"}
Jul 06 20:50:13.263156 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.263001    2569 scope.go:117] "RemoveContainer" containerID="196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"
Jul 06 20:50:13.271309 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.271292    2569 scope.go:117] "RemoveContainer" containerID="e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"
Jul 06 20:50:13.278614 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.278593    2569 scope.go:117] "RemoveContainer" containerID="b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"
Jul 06 20:50:13.285376 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.285355    2569 scope.go:117] "RemoveContainer" containerID="196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"
Jul 06 20:50:13.285616 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:13.285596    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4\": container with ID starting with 196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4 not found: ID does not exist" containerID="196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"
Jul 06 20:50:13.285686 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.285626    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4"} err="failed to get container status \"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4\": rpc error: code = NotFound desc = could not find container \"196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4\": container with ID starting with 196f976440b4080218a1a0c1b0faf70e480acd06b8e63bad00b70ba6e854ffe4 not found: ID does not exist"
Jul 06 20:50:13.285686 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.285645    2569 scope.go:117] "RemoveContainer" containerID="e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"
Jul 06 20:50:13.286409 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:13.286346    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f\": container with ID starting with e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f not found: ID does not exist" containerID="e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"
Jul 06 20:50:13.286409 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.286379    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f"} err="failed to get container status \"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f\": rpc error: code = NotFound desc = could not find container \"e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f\": container with ID starting with e7522d8ef2f743f74107de8dc8088eb5a77a2cdc5e9033863d1d70e18905041f not found: ID does not exist"
Jul 06 20:50:13.286409 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.286401    2569 scope.go:117] "RemoveContainer" containerID="b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"
Jul 06 20:50:13.286667 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:50:13.286651    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618\": container with ID starting with b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618 not found: ID does not exist" containerID="b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"
Jul 06 20:50:13.286708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.286671    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618"} err="failed to get container status \"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618\": rpc error: code = NotFound desc = could not find container \"b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618\": container with ID starting with b0651734df57917431570f6a6ac7d9337abf337158c2841c3b9f12c8f4b97618 not found: ID does not exist"
Jul 06 20:50:13.287910 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.287893    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:50:13.296925 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.296906    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn"]
Jul 06 20:50:13.826363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.826324    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" path="/var/lib/kubelet/pods/834395f7-75f2-4a3c-a0fa-e8d3e1552e05/volumes"
Jul 06 20:50:13.826777 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:13.826764    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" path="/var/lib/kubelet/pods/fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05/volumes"
Jul 06 20:50:17.262592 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.262564    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:50:17.723280 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.723246    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.827038 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.826997    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:50:17.827322 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827311    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827325    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827334    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827339    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="storage-initializer"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827352    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="storage-initializer"
Jul 06 20:50:17.827363 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="storage-initializer"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827364    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="storage-initializer"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827379    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827385    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827393    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827397    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827446    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827456    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa27e854-5a03-43bf-b4d9-bc2f9fcf1d05" containerName="kserve-container"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827467    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kserve-container"
Jul 06 20:50:17.827532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.827474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="834395f7-75f2-4a3c-a0fa-e8d3e1552e05" containerName="kube-rbac-proxy"
Jul 06 20:50:17.831952 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.831933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.835802 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.835782    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:50:17.836129 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.836098    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-e7cb2-kube-rbac-proxy-sar-config\""
Jul 06 20:50:17.836129 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.836125    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-e7cb2-predictor-serving-cert\""
Jul 06 20:50:17.836291 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.836162    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:50:17.840832 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.840809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.840930 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.840843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.840930 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.840873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdv6k\" (UniqueName: \"kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.841044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.840959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.841044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.840986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.852811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.852779    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:50:17.905395 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.905362    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.941501 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.941708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.941708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.941708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.941708 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdv6k\" (UniqueName: \"kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.942024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.941983    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.942340 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.942289    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.942340 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.942361    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.944096 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.944077    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:17.952489 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:17.952469    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdv6k\" (UniqueName: \"kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k\") pod \"isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:18.141602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:18.141555    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:18.280150 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:18.280046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:50:18.282419 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:50:18.282388    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ba0a95_139b_4d29_9b87_2edebbcae931.slice/crio-fb78c2c3a10cba96f14921f498f0cadbbff61cf02a4063721399708bbf40e04c WatchSource:0}: Error finding container fb78c2c3a10cba96f14921f498f0cadbbff61cf02a4063721399708bbf40e04c: Status 404 returned error can't find the container with id fb78c2c3a10cba96f14921f498f0cadbbff61cf02a4063721399708bbf40e04c
Jul 06 20:50:19.287994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:19.287959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerStarted","Data":"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"}
Jul 06 20:50:19.288385 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:19.288034    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerStarted","Data":"fb78c2c3a10cba96f14921f498f0cadbbff61cf02a4063721399708bbf40e04c"}
Jul 06 20:50:22.300394 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:22.300362    2569 generic.go:356] "Generic (PLEG): container finished" podID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerID="7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79" exitCode=0
Jul 06 20:50:22.300817 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:22.300422    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerDied","Data":"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"}
Jul 06 20:50:23.306833 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.306798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerStarted","Data":"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"}
Jul 06 20:50:23.307267 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.306864    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerStarted","Data":"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"}
Jul 06 20:50:23.307267 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.306890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerStarted","Data":"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"}
Jul 06 20:50:23.307418 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.307341    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:23.307418 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.307392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:23.307418 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.307415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:23.308734 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.308703    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:23.309439 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.309420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:23.340608 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:23.340557    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podStartSLOduration=2.721327231 podStartE2EDuration="6.340543891s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61921666s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:23.340104687 +0000 UTC m=+918.070670195" watchObservedRunningTime="2026-07-06 20:50:23.340543891 +0000 UTC m=+918.071109375"
Jul 06 20:50:24.311162 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:24.311122    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:24.311601 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:24.311538    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:29.315757 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:29.315730    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:50:29.316367 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:29.316343    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:29.316686 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:29.316668    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:39.316401 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:39.316365    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:39.316876 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:39.316850    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:49.316371 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:49.316322    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:49.316939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:49.316771    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:59.317116 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:59.317069    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:50:59.317626 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:50:59.317601    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:09.316710 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:09.316663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:51:09.317192 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:09.317086    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:19.317217 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:19.317179    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:51:19.317759 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:19.317736    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:29.316604 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:29.316507    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:51:29.317089 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:29.317056    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:39.316962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:39.316934    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:51:39.317482 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:39.317352    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:51:39.792946 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:39.792907    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:52.732241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:52.732205    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-e7cb2-predictor-689764df45-x5zcv_1bdbbb2c-7909-4561-9234-211d7d74223b/kserve-container/0.log"
Jul 06 20:51:53.047998 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.047950    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.137607 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.137574    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:51:53.137884 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.137835    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" containerID="cri-o://be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a" gracePeriod=30
Jul 06 20:51:53.137962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.137886    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" containerID="cri-o://174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc" gracePeriod=30
Jul 06 20:51:53.137962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.137865    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" containerID="cri-o://7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0" gracePeriod=30
Jul 06 20:51:53.141509 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.141350    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:51:53.145881 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.145857    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.148173 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.148128    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.148952 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.148930    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-26dd6-kube-rbac-proxy-sar-config\""
Jul 06 20:51:53.149085 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.148982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-26dd6-predictor-serving-cert\""
Jul 06 20:51:53.159970 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.159941    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:51:53.161831 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.161805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.161992 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.161847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.161992 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.161900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.161992 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.161944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fk7\" (UniqueName: \"kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.161992 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.161982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.195602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.195569    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:51:53.195829 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.195791    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kserve-container" containerID="cri-o://0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa" gracePeriod=30
Jul 06 20:51:53.195907 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.195879    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kube-rbac-proxy" containerID="cri-o://ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704" gracePeriod=30
Jul 06 20:51:53.225576 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.225547    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:53.262658 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.262632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.262837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.262664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.262837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.262727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.262837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.262773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fk7\" (UniqueName: \"kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.262837 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.262811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.263076 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.262865    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-serving-cert: secret "isvc-sklearn-scale-raw-26dd6-predictor-serving-cert" not found
Jul 06 20:51:53.263076 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.262942    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls podName:e737beeb-91f7-4517-9449-72b17679b380 nodeName:}" failed. No retries permitted until 2026-07-06 20:51:53.762918979 +0000 UTC m=+1008.493484460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls") pod "isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" (UID: "e737beeb-91f7-4517-9449-72b17679b380") : secret "isvc-sklearn-scale-raw-26dd6-predictor-serving-cert" not found
Jul 06 20:51:53.263206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.263185    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.263463 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.263442    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.263501 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.263442    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.273743 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.273719    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fk7\" (UniqueName: \"kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.440309 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.440284    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:51:53.464809 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.464774    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config\") pod \"1bdbbb2c-7909-4561-9234-211d7d74223b\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") "
Jul 06 20:51:53.464995 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.464841    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls\") pod \"1bdbbb2c-7909-4561-9234-211d7d74223b\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") "
Jul 06 20:51:53.464995 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.464886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sdks\" (UniqueName: \"kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks\") pod \"1bdbbb2c-7909-4561-9234-211d7d74223b\" (UID: \"1bdbbb2c-7909-4561-9234-211d7d74223b\") "
Jul 06 20:51:53.465184 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.465161    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1bdbbb2c-7909-4561-9234-211d7d74223b" (UID: "1bdbbb2c-7909-4561-9234-211d7d74223b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:51:53.466979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.466957    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1bdbbb2c-7909-4561-9234-211d7d74223b" (UID: "1bdbbb2c-7909-4561-9234-211d7d74223b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:51:53.466979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.466966    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks" (OuterVolumeSpecName: "kube-api-access-7sdks") pod "1bdbbb2c-7909-4561-9234-211d7d74223b" (UID: "1bdbbb2c-7909-4561-9234-211d7d74223b"). InnerVolumeSpecName "kube-api-access-7sdks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:51:53.566099 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.565981    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdbbb2c-7909-4561-9234-211d7d74223b-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:51:53.566099 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.566040    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdks\" (UniqueName: \"kubernetes.io/projected/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-api-access-7sdks\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:51:53.566099 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.566054    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdbbb2c-7909-4561-9234-211d7d74223b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:51:53.647933 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.647888    2569 generic.go:356] "Generic (PLEG): container finished" podID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerID="ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704" exitCode=2
Jul 06 20:51:53.647933 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.647915    2569 generic.go:356] "Generic (PLEG): container finished" podID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerID="0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa" exitCode=2
Jul 06 20:51:53.647933 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.647941    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"
Jul 06 20:51:53.648206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.647972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerDied","Data":"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"}
Jul 06 20:51:53.648206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.648051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerDied","Data":"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"}
Jul 06 20:51:53.648206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.648076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv" event={"ID":"1bdbbb2c-7909-4561-9234-211d7d74223b","Type":"ContainerDied","Data":"89cb48dd8ee750f6b0ccd769f69c476e68413040fcae7b13bd299faee296c67d"}
Jul 06 20:51:53.648206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.648111    2569 scope.go:117] "RemoveContainer" containerID="ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"
Jul 06 20:51:53.650844 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.650818    2569 generic.go:356] "Generic (PLEG): container finished" podID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerID="174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc" exitCode=2
Jul 06 20:51:53.650958 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.650888    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerDied","Data":"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"}
Jul 06 20:51:53.656585 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.656564    2569 scope.go:117] "RemoveContainer" containerID="0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"
Jul 06 20:51:53.664082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664064    2569 scope.go:117] "RemoveContainer" containerID="ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"
Jul 06 20:51:53.664334 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.664316    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704\": container with ID starting with ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704 not found: ID does not exist" containerID="ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"
Jul 06 20:51:53.664387 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664343    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"} err="failed to get container status \"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704\": rpc error: code = NotFound desc = could not find container \"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704\": container with ID starting with ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704 not found: ID does not exist"
Jul 06 20:51:53.664387 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664359    2569 scope.go:117] "RemoveContainer" containerID="0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"
Jul 06 20:51:53.664591 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.664572    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa\": container with ID starting with 0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa not found: ID does not exist" containerID="0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"
Jul 06 20:51:53.664655 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664599    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"} err="failed to get container status \"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa\": rpc error: code = NotFound desc = could not find container \"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa\": container with ID starting with 0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa not found: ID does not exist"
Jul 06 20:51:53.664655 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664623    2569 scope.go:117] "RemoveContainer" containerID="ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"
Jul 06 20:51:53.664840 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664815    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704"} err="failed to get container status \"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704\": rpc error: code = NotFound desc = could not find container \"ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704\": container with ID starting with ffc5bef3005b547247d24006ff6f75890fdc27e532d97743996ecc0cf02e7704 not found: ID does not exist"
Jul 06 20:51:53.664883 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.664841    2569 scope.go:117] "RemoveContainer" containerID="0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"
Jul 06 20:51:53.665041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.665025    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa"} err="failed to get container status \"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa\": rpc error: code = NotFound desc = could not find container \"0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa\": container with ID starting with 0d8575828d8d5cf1be9bb6737ed1d23c4762387b05298dcad1246376f34a85fa not found: ID does not exist"
Jul 06 20:51:53.672791 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.672769    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:51:53.686736 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.686715    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv"]
Jul 06 20:51:53.768082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.768044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:53.768460 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.768190    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-serving-cert: secret "isvc-sklearn-scale-raw-26dd6-predictor-serving-cert" not found
Jul 06 20:51:53.768460 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:51:53.768246    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls podName:e737beeb-91f7-4517-9449-72b17679b380 nodeName:}" failed. No retries permitted until 2026-07-06 20:51:54.768231482 +0000 UTC m=+1009.498796945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls") pod "isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" (UID: "e737beeb-91f7-4517-9449-72b17679b380") : secret "isvc-sklearn-scale-raw-26dd6-predictor-serving-cert" not found
Jul 06 20:51:53.826637 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:53.826560    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" path="/var/lib/kubelet/pods/1bdbbb2c-7909-4561-9234-211d7d74223b/volumes"
Jul 06 20:51:54.053229 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:54.053190    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:54.311603 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:54.311564    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:51:54.777192 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:54.777155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:54.779669 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:54.779640    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:54.963359 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:54.963320    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:51:55.107614 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:55.107557    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:51:55.109852 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:51:55.109824    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode737beeb_91f7_4517_9449_72b17679b380.slice/crio-3b4e375497f259e148ca30de3cbc169b29d2cb31355ed3f769a035515b67747b WatchSource:0}: Error finding container 3b4e375497f259e148ca30de3cbc169b29d2cb31355ed3f769a035515b67747b: Status 404 returned error can't find the container with id 3b4e375497f259e148ca30de3cbc169b29d2cb31355ed3f769a035515b67747b
Jul 06 20:51:55.661526 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:55.661489    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerStarted","Data":"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"}
Jul 06 20:51:55.661728 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:55.661556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerStarted","Data":"3b4e375497f259e148ca30de3cbc169b29d2cb31355ed3f769a035515b67747b"}
Jul 06 20:51:58.675387 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:58.675356    2569 generic.go:356] "Generic (PLEG): container finished" podID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerID="be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a" exitCode=0
Jul 06 20:51:58.675762 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:58.675427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerDied","Data":"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"}
Jul 06 20:51:59.311394 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:59.311351    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:51:59.317241 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:59.317211    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:59.317389 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:59.317368    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:51:59.680575 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:59.680496    2569 generic.go:356] "Generic (PLEG): container finished" podID="e737beeb-91f7-4517-9449-72b17679b380" containerID="9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece" exitCode=0
Jul 06 20:51:59.680575 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:51:59.680557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerDied","Data":"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"}
Jul 06 20:52:00.686425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:00.686385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerStarted","Data":"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"}
Jul 06 20:52:00.686814 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:00.686447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerStarted","Data":"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"}
Jul 06 20:52:00.686814 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:00.686628    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:52:00.718063 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:00.717998    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podStartSLOduration=3.987502723 podStartE2EDuration="7.717984527s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.730481804s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:52:00.714583069 +0000 UTC m=+1015.445148554" watchObservedRunningTime="2026-07-06 20:52:00.717984527 +0000 UTC m=+1015.448550011"
Jul 06 20:52:01.690471 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:01.690439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:52:01.691769 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:01.691737    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:02.693697 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:02.693664    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:04.311515 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:04.311471    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:52:04.311954 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:04.311621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:52:07.699026 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:07.698977    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:52:07.699513 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:07.699487    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:09.311383 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:09.311343    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:52:09.317093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:09.317063    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:09.318222 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:09.318197    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:52:09.535918 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:09.535882    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:14.311347 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:14.311302    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:52:17.700018 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:17.699967    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:19.311525 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:19.311489    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 06 20:52:19.317386 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:19.317348    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 06 20:52:19.317496 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:19.317481    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:52:19.317539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:19.317493    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:19.317595 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:19.317583    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:52:23.300113 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.300088    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:52:23.411636 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411545    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config\") pod \"a5ba0a95-139b-4d29-9b87-2edebbcae931\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") "
Jul 06 20:52:23.411636 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411595    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls\") pod \"a5ba0a95-139b-4d29-9b87-2edebbcae931\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") "
Jul 06 20:52:23.411636 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411621    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdv6k\" (UniqueName: \"kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k\") pod \"a5ba0a95-139b-4d29-9b87-2edebbcae931\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") "
Jul 06 20:52:23.411636 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411640    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location\") pod \"a5ba0a95-139b-4d29-9b87-2edebbcae931\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") "
Jul 06 20:52:23.411970 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert\") pod \"a5ba0a95-139b-4d29-9b87-2edebbcae931\" (UID: \"a5ba0a95-139b-4d29-9b87-2edebbcae931\") "
Jul 06 20:52:23.411970 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411926    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5ba0a95-139b-4d29-9b87-2edebbcae931" (UID: "a5ba0a95-139b-4d29-9b87-2edebbcae931"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:52:23.411970 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.411937    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5ba0a95-139b-4d29-9b87-2edebbcae931" (UID: "a5ba0a95-139b-4d29-9b87-2edebbcae931"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:23.412160 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.412141    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5ba0a95-139b-4d29-9b87-2edebbcae931" (UID: "a5ba0a95-139b-4d29-9b87-2edebbcae931"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:23.412194 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.412145    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5ba0a95-139b-4d29-9b87-2edebbcae931-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:52:23.412194 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.412181    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:52:23.413962 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.413939    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k" (OuterVolumeSpecName: "kube-api-access-tdv6k") pod "a5ba0a95-139b-4d29-9b87-2edebbcae931" (UID: "a5ba0a95-139b-4d29-9b87-2edebbcae931"). InnerVolumeSpecName "kube-api-access-tdv6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:52:23.414057 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.413961    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5ba0a95-139b-4d29-9b87-2edebbcae931" (UID: "a5ba0a95-139b-4d29-9b87-2edebbcae931"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:23.513676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.513636    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba0a95-139b-4d29-9b87-2edebbcae931-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:52:23.513676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.513666    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5ba0a95-139b-4d29-9b87-2edebbcae931-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:52:23.513676 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.513675    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdv6k\" (UniqueName: \"kubernetes.io/projected/a5ba0a95-139b-4d29-9b87-2edebbcae931-kube-api-access-tdv6k\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:52:23.774792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.774716    2569 generic.go:356] "Generic (PLEG): container finished" podID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerID="7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0" exitCode=0
Jul 06 20:52:23.774792 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.774781    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"
Jul 06 20:52:23.774994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.774815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerDied","Data":"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"}
Jul 06 20:52:23.774994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.774899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw" event={"ID":"a5ba0a95-139b-4d29-9b87-2edebbcae931","Type":"ContainerDied","Data":"fb78c2c3a10cba96f14921f498f0cadbbff61cf02a4063721399708bbf40e04c"}
Jul 06 20:52:23.774994 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.774945    2569 scope.go:117] "RemoveContainer" containerID="7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"
Jul 06 20:52:23.783641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.783624    2569 scope.go:117] "RemoveContainer" containerID="174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"
Jul 06 20:52:23.791163 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.791147    2569 scope.go:117] "RemoveContainer" containerID="be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"
Jul 06 20:52:23.798292 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.798266    2569 scope.go:117] "RemoveContainer" containerID="7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"
Jul 06 20:52:23.805366 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.805347    2569 scope.go:117] "RemoveContainer" containerID="7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"
Jul 06 20:52:23.805636 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:52:23.805619    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0\": container with ID starting with 7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0 not found: ID does not exist" containerID="7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"
Jul 06 20:52:23.805687 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.805645    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0"} err="failed to get container status \"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0\": rpc error: code = NotFound desc = could not find container \"7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0\": container with ID starting with 7079d4b5efe71efc1b2f6bad627d9d4eae9bcd559b73240ee51cca4e70a550f0 not found: ID does not exist"
Jul 06 20:52:23.805687 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.805665    2569 scope.go:117] "RemoveContainer" containerID="174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"
Jul 06 20:52:23.805950 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:52:23.805933    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc\": container with ID starting with 174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc not found: ID does not exist" containerID="174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"
Jul 06 20:52:23.806093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.805959    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc"} err="failed to get container status \"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc\": rpc error: code = NotFound desc = could not find container \"174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc\": container with ID starting with 174bf2793f500ccd3c53d16a3d3728938dfd75b9ac781b1ed88337ba1124afcc not found: ID does not exist"
Jul 06 20:52:23.806093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.805982    2569 scope.go:117] "RemoveContainer" containerID="be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"
Jul 06 20:52:23.806264 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:52:23.806245    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a\": container with ID starting with be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a not found: ID does not exist" containerID="be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"
Jul 06 20:52:23.806319 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.806272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a"} err="failed to get container status \"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a\": rpc error: code = NotFound desc = could not find container \"be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a\": container with ID starting with be0b548f6520bb714a2374ad72eee18a7cc28854e8a5ba695a9843145608b11a not found: ID does not exist"
Jul 06 20:52:23.806319 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.806295    2569 scope.go:117] "RemoveContainer" containerID="7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"
Jul 06 20:52:23.806561 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:52:23.806543    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79\": container with ID starting with 7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79 not found: ID does not exist" containerID="7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"
Jul 06 20:52:23.806597 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.806567    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79"} err="failed to get container status \"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79\": rpc error: code = NotFound desc = could not find container \"7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79\": container with ID starting with 7f56ea11881d77c5543b9ab6e79dcb20874c11d7fe4f02ab03e9d52d9dbeca79 not found: ID does not exist"
Jul 06 20:52:23.813559 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.813492    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:52:23.814718 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.814697    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw"]
Jul 06 20:52:23.826001 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:23.825978    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" path="/var/lib/kubelet/pods/a5ba0a95-139b-4d29-9b87-2edebbcae931/volumes"
Jul 06 20:52:27.699453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:27.699415    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:37.699605 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:37.699562    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:47.699939 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:47.699899    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:52:57.699828 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:52:57.699738    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:07.699824 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:07.699783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:14.822474 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:14.822427    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:24.822751 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:24.822705    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:34.822532 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:34.822479    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:44.823157 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:44.823114    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:53:54.822347 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:53:54.822307    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:54:04.822614 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:04.822573    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:54:14.822396 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:14.822355    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:54:23.822041 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:23.821973    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:54:33.826547 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:33.826477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:54:43.270693 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:43.270660    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:54:43.271260 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:43.270965    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" containerID="cri-o://aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02" gracePeriod=30
Jul 06 20:54:43.271260 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:43.270967    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy" containerID="cri-o://ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e" gracePeriod=30
Jul 06 20:54:43.311170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:43.311134    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:43.822536 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:43.822498    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 06 20:54:44.278232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:44.278149    2569 generic.go:356] "Generic (PLEG): container finished" podID="e737beeb-91f7-4517-9449-72b17679b380" containerID="ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e" exitCode=2
Jul 06 20:54:44.278232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:44.278202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerDied","Data":"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"}
Jul 06 20:54:47.694428 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:47.694387    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 06 20:54:48.339640 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.339600    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:48.414104 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414057    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:54:48.414371 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414359    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414372    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414383    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414388    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414394    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="storage-initializer"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414399    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="storage-initializer"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414406    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent"
Jul 06 20:54:48.414415 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414411    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414426    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414431    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414440    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kserve-container"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414446    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kserve-container"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414489    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414497    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kube-rbac-proxy"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414504    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="agent"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414510    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ba0a95-139b-4d29-9b87-2edebbcae931" containerName="kserve-container"
Jul 06 20:54:48.414609 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.414515    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdbbb2c-7909-4561-9234-211d7d74223b" containerName="kserve-container"
Jul 06 20:54:48.417541 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.417525    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.420206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.420179    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f8d41a-predictor-serving-cert\""
Jul 06 20:54:48.420432 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.420413    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f8d41a-kube-rbac-proxy-sar-config\""
Jul 06 20:54:48.437733 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.437708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:54:48.601863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.601794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbk9j\" (UniqueName: \"kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.601863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.601825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.601863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.601852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.602093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.601899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.602093 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.601941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.628225 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.628191    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:48.702997 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.702961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbk9j\" (UniqueName: \"kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.702996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703441    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703795 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703776    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.703889 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.703868    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.705679 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.705663    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.712345 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.712321    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbk9j\" (UniqueName: \"kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j\") pod \"isvc-primary-f8d41a-predictor-656db64bb8-495pz\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") " pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.727066 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.727041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:48.855305 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:48.855281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:54:48.857611 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:54:48.857571    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2badce2e_d8c6_408c_8b4a_e55b3f7d3dc1.slice/crio-45cff3c3b055b492debe73e2f75dc2e5943cbcf9b1a9a0524197851e6ed9e45a WatchSource:0}: Error finding container 45cff3c3b055b492debe73e2f75dc2e5943cbcf9b1a9a0524197851e6ed9e45a: Status 404 returned error can't find the container with id 45cff3c3b055b492debe73e2f75dc2e5943cbcf9b1a9a0524197851e6ed9e45a
Jul 06 20:54:49.298114 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:49.298085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerStarted","Data":"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"}
Jul 06 20:54:49.298258 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:49.298143    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerStarted","Data":"45cff3c3b055b492debe73e2f75dc2e5943cbcf9b1a9a0524197851e6ed9e45a"}
Jul 06 20:54:52.694438 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:52.694396    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 06 20:54:53.005930 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.005910    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:54:53.136746 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.136713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert\") pod \"e737beeb-91f7-4517-9449-72b17679b380\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") "
Jul 06 20:54:53.137495 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.137461    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e737beeb-91f7-4517-9449-72b17679b380" (UID: "e737beeb-91f7-4517-9449-72b17679b380"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:54:53.137726 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.137709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2fk7\" (UniqueName: \"kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7\") pod \"e737beeb-91f7-4517-9449-72b17679b380\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") "
Jul 06 20:54:53.137866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.137853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location\") pod \"e737beeb-91f7-4517-9449-72b17679b380\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") "
Jul 06 20:54:53.138032 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.137997    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") pod \"e737beeb-91f7-4517-9449-72b17679b380\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") "
Jul 06 20:54:53.138164 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.138151    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config\") pod \"e737beeb-91f7-4517-9449-72b17679b380\" (UID: \"e737beeb-91f7-4517-9449-72b17679b380\") "
Jul 06 20:54:53.138556 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.138540    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:54:53.139044 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.139023    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e737beeb-91f7-4517-9449-72b17679b380" (UID: "e737beeb-91f7-4517-9449-72b17679b380"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:54:53.139466 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.139448    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e737beeb-91f7-4517-9449-72b17679b380" (UID: "e737beeb-91f7-4517-9449-72b17679b380"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:53.143042 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.142944    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e737beeb-91f7-4517-9449-72b17679b380" (UID: "e737beeb-91f7-4517-9449-72b17679b380"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:54:53.143042 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.142950    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7" (OuterVolumeSpecName: "kube-api-access-v2fk7") pod "e737beeb-91f7-4517-9449-72b17679b380" (UID: "e737beeb-91f7-4517-9449-72b17679b380"). InnerVolumeSpecName "kube-api-access-v2fk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:54:53.239695 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.239623    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e737beeb-91f7-4517-9449-72b17679b380-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:54:53.239695 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.239650    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e737beeb-91f7-4517-9449-72b17679b380-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:54:53.239695 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.239660    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e737beeb-91f7-4517-9449-72b17679b380-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:54:53.239695 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.239669    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fk7\" (UniqueName: \"kubernetes.io/projected/e737beeb-91f7-4517-9449-72b17679b380-kube-api-access-v2fk7\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:54:53.314864 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.314833    2569 generic.go:356] "Generic (PLEG): container finished" podID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerID="22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898" exitCode=0
Jul 06 20:54:53.315061 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.314908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerDied","Data":"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"}
Jul 06 20:54:53.316766 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.316742    2569 generic.go:356] "Generic (PLEG): container finished" podID="e737beeb-91f7-4517-9449-72b17679b380" containerID="aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02" exitCode=0
Jul 06 20:54:53.316867 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.316783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerDied","Data":"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"}
Jul 06 20:54:53.316867 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.316812    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"
Jul 06 20:54:53.316867 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.316825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg" event={"ID":"e737beeb-91f7-4517-9449-72b17679b380","Type":"ContainerDied","Data":"3b4e375497f259e148ca30de3cbc169b29d2cb31355ed3f769a035515b67747b"}
Jul 06 20:54:53.316867 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.316850    2569 scope.go:117] "RemoveContainer" containerID="ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"
Jul 06 20:54:53.326909 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.326888    2569 scope.go:117] "RemoveContainer" containerID="aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"
Jul 06 20:54:53.335421 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.335398    2569 scope.go:117] "RemoveContainer" containerID="9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"
Jul 06 20:54:53.351246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.350992    2569 scope.go:117] "RemoveContainer" containerID="ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"
Jul 06 20:54:53.351246 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351224    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:54:53.351404 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:54:53.351345    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e\": container with ID starting with ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e not found: ID does not exist" containerID="ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"
Jul 06 20:54:53.351404 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351377    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e"} err="failed to get container status \"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e\": rpc error: code = NotFound desc = could not find container \"ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e\": container with ID starting with ac9de671f3a112ef0d4983da0a8ee6255171551bb0567499575704416846609e not found: ID does not exist"
Jul 06 20:54:53.351404 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351395    2569 scope.go:117] "RemoveContainer" containerID="aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"
Jul 06 20:54:53.351668 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:54:53.351650    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02\": container with ID starting with aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02 not found: ID does not exist" containerID="aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"
Jul 06 20:54:53.351723 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351672    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02"} err="failed to get container status \"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02\": rpc error: code = NotFound desc = could not find container \"aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02\": container with ID starting with aa6f581f02f331f5b3025e7744efe410a52c6c0aed2a6121c79d7ccd171fdf02 not found: ID does not exist"
Jul 06 20:54:53.351723 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351688    2569 scope.go:117] "RemoveContainer" containerID="9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"
Jul 06 20:54:53.351979 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:54:53.351962    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece\": container with ID starting with 9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece not found: ID does not exist" containerID="9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"
Jul 06 20:54:53.352074 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.351987    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece"} err="failed to get container status \"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece\": rpc error: code = NotFound desc = could not find container \"9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece\": container with ID starting with 9d4dd06babdd58d1a5395de0986c4b4cf65e8f740ba20f226d90cd07bb37aece not found: ID does not exist"
Jul 06 20:54:53.355712 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.355690    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg"]
Jul 06 20:54:53.826702 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:53.826669    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e737beeb-91f7-4517-9449-72b17679b380" path="/var/lib/kubelet/pods/e737beeb-91f7-4517-9449-72b17679b380/volumes"
Jul 06 20:54:54.322342 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:54.322314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerStarted","Data":"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"}
Jul 06 20:54:54.322511 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:54.322381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerStarted","Data":"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"}
Jul 06 20:54:54.322624 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:54.322595    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:54.344425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:54.344374    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podStartSLOduration=2.454324614 podStartE2EDuration="6.34435956s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.890034946s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:54.342939369 +0000 UTC m=+1189.073504854" watchObservedRunningTime="2026-07-06 20:54:54.34435956 +0000 UTC m=+1189.074925044"
Jul 06 20:54:55.327076 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:55.327038    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:54:55.328158 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:55.328131    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:54:56.330272 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:54:56.330233    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:01.335968 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:01.335939    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:55:01.336500 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:01.336473    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:11.336682 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:11.336643    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:21.337071 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:21.337034    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:31.336392 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:31.336351    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:41.336565 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:41.336518    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:55:51.336889 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:55:51.336846    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:56:01.337124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:01.337034    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:56:11.337165 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:11.337134    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:56:18.542269 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.542233    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:18.617340 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617301    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:18.617631 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617619    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container"
Jul 06 20:56:18.617674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617635    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container"
Jul 06 20:56:18.617674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy"
Jul 06 20:56:18.617674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617655    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy"
Jul 06 20:56:18.617674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617665    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="storage-initializer"
Jul 06 20:56:18.617674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617670    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="storage-initializer"
Jul 06 20:56:18.617820 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617720    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kserve-container"
Jul 06 20:56:18.617820 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.617735    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e737beeb-91f7-4517-9449-72b17679b380" containerName="kube-rbac-proxy"
Jul 06 20:56:18.620938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.620914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.623945 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.623912    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f8d41a-predictor-serving-cert\""
Jul 06 20:56:18.624082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.623918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f8d41a-kube-rbac-proxy-sar-config\""
Jul 06 20:56:18.624082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.624001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f8d41a-dockercfg-nvq7f\""
Jul 06 20:56:18.624082 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.624063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f8d41a\""
Jul 06 20:56:18.631704 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.631680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:18.689209 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.689173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.689209 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.689208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qlq\" (UniqueName: \"kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.689442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.689231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.689442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.689307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.689442 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.689339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.768405 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.768370    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:18.790443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.790443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.790701 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.790701 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qlq\" (UniqueName: \"kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.790701 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.790967 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.790943    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.791148 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.791130    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.791227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.791196    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.793085 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.793028    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.801515 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.801493    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qlq\" (UniqueName: \"kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq\") pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") " pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:18.932639 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:18.932600    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:19.071123 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:19.071037    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:19.073807 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:56:19.073774    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e6f04f_745d_4ace_8dd1_477f950a7aaf.slice/crio-ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d WatchSource:0}: Error finding container ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d: Status 404 returned error can't find the container with id ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d
Jul 06 20:56:19.636961 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:19.636918    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerStarted","Data":"00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c"}
Jul 06 20:56:19.637433 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:19.636988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerStarted","Data":"ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d"}
Jul 06 20:56:21.646420 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:21.646344    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/0.log"
Jul 06 20:56:21.646420 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:21.646385    2569 generic.go:356] "Generic (PLEG): container finished" podID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerID="00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c" exitCode=1
Jul 06 20:56:21.646816 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:21.646437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerDied","Data":"00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c"}
Jul 06 20:56:22.651619 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:22.651587    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/0.log"
Jul 06 20:56:22.652232 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:22.651695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerStarted","Data":"6f4a6fe19dc7d7dc59eaf3329497dcb846c7a3b3b3b758f6e22da29f3c863953"}
Jul 06 20:56:24.661356 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.661326    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/1.log"
Jul 06 20:56:24.661772 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.661705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/0.log"
Jul 06 20:56:24.661772 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.661738    2569 generic.go:356] "Generic (PLEG): container finished" podID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerID="6f4a6fe19dc7d7dc59eaf3329497dcb846c7a3b3b3b758f6e22da29f3c863953" exitCode=1
Jul 06 20:56:24.661847 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.661791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerDied","Data":"6f4a6fe19dc7d7dc59eaf3329497dcb846c7a3b3b3b758f6e22da29f3c863953"}
Jul 06 20:56:24.661882 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.661846    2569 scope.go:117] "RemoveContainer" containerID="00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c"
Jul 06 20:56:24.662290 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:24.662275    2569 scope.go:117] "RemoveContainer" containerID="00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c"
Jul 06 20:56:24.673711 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:24.673681    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_kserve-ci-e2e-test_47e6f04f-745d-4ace-8dd1-477f950a7aaf_0 in pod sandbox ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d from index: no such id: '00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c'" containerID="00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c"
Jul 06 20:56:24.673928 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:24.673890    2569 kuberuntime_container.go:961] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_kserve-ci-e2e-test_47e6f04f-745d-4ace-8dd1-477f950a7aaf_0 in pod sandbox ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d from index: no such id: '00e42e1b22c7c079f4836002ddc769c8fbbb8fd0bf12f6def304575563b3225c'; Skipping pod \"isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_kserve-ci-e2e-test(47e6f04f-745d-4ace-8dd1-477f950a7aaf)\"" logger="UnhandledError"
Jul 06 20:56:24.674186 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:24.674161    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_kserve-ci-e2e-test(47e6f04f-745d-4ace-8dd1-477f950a7aaf)\"" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf"
Jul 06 20:56:25.667724 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:25.667693    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/1.log"
Jul 06 20:56:32.706458 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.706421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:32.723998 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.723963    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:32.774981 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.774949    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:56:32.775266 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.775221    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container" containerID="cri-o://7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519" gracePeriod=30
Jul 06 20:56:32.775417 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.775250    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kube-rbac-proxy" containerID="cri-o://fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7" gracePeriod=30
Jul 06 20:56:32.796056 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.796021    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:32.894782 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.894759    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/1.log"
Jul 06 20:56:32.896938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.896923    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:32.994953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.994859    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6qlq\" (UniqueName: \"kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq\") pod \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") "
Jul 06 20:56:32.994953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.994926    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config\") pod \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") "
Jul 06 20:56:32.994953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.994944    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location\") pod \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") "
Jul 06 20:56:32.995276 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995123    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls\") pod \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") "
Jul 06 20:56:32.995276 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995191    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert\") pod \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\" (UID: \"47e6f04f-745d-4ace-8dd1-477f950a7aaf\") "
Jul 06 20:56:32.995276 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995185    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47e6f04f-745d-4ace-8dd1-477f950a7aaf" (UID: "47e6f04f-745d-4ace-8dd1-477f950a7aaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:32.995434 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995387    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:32.995434 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995343    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47e6f04f-745d-4ace-8dd1-477f950a7aaf" (UID: "47e6f04f-745d-4ace-8dd1-477f950a7aaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:32.995501 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.995481    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47e6f04f-745d-4ace-8dd1-477f950a7aaf" (UID: "47e6f04f-745d-4ace-8dd1-477f950a7aaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:32.997167 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.997144    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq" (OuterVolumeSpecName: "kube-api-access-g6qlq") pod "47e6f04f-745d-4ace-8dd1-477f950a7aaf" (UID: "47e6f04f-745d-4ace-8dd1-477f950a7aaf"). InnerVolumeSpecName "kube-api-access-g6qlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:32.997296 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:32.997282    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47e6f04f-745d-4ace-8dd1-477f950a7aaf" (UID: "47e6f04f-745d-4ace-8dd1-477f950a7aaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:33.096573 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.096525    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6qlq\" (UniqueName: \"kubernetes.io/projected/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-api-access-g6qlq\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:33.096573 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.096563    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:33.096573 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.096576    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47e6f04f-745d-4ace-8dd1-477f950a7aaf-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:33.096829 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.096590    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47e6f04f-745d-4ace-8dd1-477f950a7aaf-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:33.699764 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.699730    2569 generic.go:356] "Generic (PLEG): container finished" podID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerID="fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7" exitCode=2
Jul 06 20:56:33.699959 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.699799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerDied","Data":"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"}
Jul 06 20:56:33.701016 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.700993    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_47e6f04f-745d-4ace-8dd1-477f950a7aaf/storage-initializer/1.log"
Jul 06 20:56:33.701122 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.701061    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6" event={"ID":"47e6f04f-745d-4ace-8dd1-477f950a7aaf","Type":"ContainerDied","Data":"ce1c3fe2c984f84a0a104928397d0a2e71cdfbcf03e01b54dc5229700eabca4d"}
Jul 06 20:56:33.701122 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.701091    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"
Jul 06 20:56:33.701122 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.701119    2569 scope.go:117] "RemoveContainer" containerID="6f4a6fe19dc7d7dc59eaf3329497dcb846c7a3b3b3b758f6e22da29f3c863953"
Jul 06 20:56:33.741499 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.741464    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:33.743895 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.743866    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6"]
Jul 06 20:56:33.827084 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:33.827055    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" path="/var/lib/kubelet/pods/47e6f04f-745d-4ace-8dd1-477f950a7aaf/volumes"
Jul 06 20:56:36.331162 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:36.331113    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:56:37.818425 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.818386    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:37.875075 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875036    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:37.875359 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.875406 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.875406 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875376    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.875406 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875381    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.875498 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875430    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.875539 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.875530    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e6f04f-745d-4ace-8dd1-477f950a7aaf" containerName="storage-initializer"
Jul 06 20:56:37.878829 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.878812    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.882863 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.882838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-1558a9\""
Jul 06 20:56:37.883510 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.883488    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-1558a9-kube-rbac-proxy-sar-config\""
Jul 06 20:56:37.883641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.883490    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-1558a9-dockercfg-xsqrr\""
Jul 06 20:56:37.883641 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.883629    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-1558a9-predictor-serving-cert\""
Jul 06 20:56:37.894162 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.894135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:37.930852 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.930828    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:56:37.931953 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.931933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.932077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.931963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s422l\" (UniqueName: \"kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.932077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.931989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.932227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.932128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.932227 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.932176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:37.973811 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:37.973778    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:38.032686 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032662    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls\") pod \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") "
Jul 06 20:56:38.032886 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbk9j\" (UniqueName: \"kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j\") pod \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") "
Jul 06 20:56:38.032886 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location\") pod \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") "
Jul 06 20:56:38.032886 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config\") pod \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") "
Jul 06 20:56:38.032886 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert\") pod \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\" (UID: \"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1\") "
Jul 06 20:56:38.033124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.032926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.033124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.033124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.033124 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.033330 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s422l\" (UniqueName: \"kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.033330 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033141    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" (UID: "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:38.033330 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033243    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" (UID: "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:38.033330 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.033242    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" (UID: "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:38.034061 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.034036    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.034198 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.034177    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.034411 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.034387    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.035235 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.035210    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j" (OuterVolumeSpecName: "kube-api-access-tbk9j") pod "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" (UID: "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1"). InnerVolumeSpecName "kube-api-access-tbk9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:38.035334 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.035253    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" (UID: "2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:38.035700 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.035683    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.043798 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.043773    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s422l\" (UniqueName: \"kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l\") pod \"isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.133524 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.133490    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:38.133524 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.133520    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbk9j\" (UniqueName: \"kubernetes.io/projected/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-api-access-tbk9j\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:38.133524 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.133530    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:38.133777 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.133539    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:38.133777 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.133547    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:38.189808 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.189766    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:38.325337 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.325226    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:38.327792 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:56:38.327772    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1d570e_cd70_45f3_b7ea_1ff67e824ce0.slice/crio-678ef445f871f39e2560dff53e013676c239207c9a4da5649559c87bb1445c0c WatchSource:0}: Error finding container 678ef445f871f39e2560dff53e013676c239207c9a4da5649559c87bb1445c0c: Status 404 returned error can't find the container with id 678ef445f871f39e2560dff53e013676c239207c9a4da5649559c87bb1445c0c
Jul 06 20:56:38.723077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.722973    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerStarted","Data":"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"}
Jul 06 20:56:38.723077 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.723054    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerStarted","Data":"678ef445f871f39e2560dff53e013676c239207c9a4da5649559c87bb1445c0c"}
Jul 06 20:56:38.724975 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.724945    2569 generic.go:356] "Generic (PLEG): container finished" podID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerID="7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519" exitCode=0
Jul 06 20:56:38.725159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.724991    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"
Jul 06 20:56:38.725159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.725041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerDied","Data":"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"}
Jul 06 20:56:38.725159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.725109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz" event={"ID":"2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1","Type":"ContainerDied","Data":"45cff3c3b055b492debe73e2f75dc2e5943cbcf9b1a9a0524197851e6ed9e45a"}
Jul 06 20:56:38.725159 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.725149    2569 scope.go:117] "RemoveContainer" containerID="fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"
Jul 06 20:56:38.734782 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.734758    2569 scope.go:117] "RemoveContainer" containerID="7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"
Jul 06 20:56:38.743642 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.743619    2569 scope.go:117] "RemoveContainer" containerID="22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"
Jul 06 20:56:38.752561 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.752536    2569 scope.go:117] "RemoveContainer" containerID="fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"
Jul 06 20:56:38.752970 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:38.752940    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7\": container with ID starting with fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7 not found: ID does not exist" containerID="fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"
Jul 06 20:56:38.753108 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.752980    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7"} err="failed to get container status \"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7\": rpc error: code = NotFound desc = could not find container \"fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7\": container with ID starting with fae3e0ab96bfd9059b08158a22534e070ff3d3db33838bab2f40271a4c815af7 not found: ID does not exist"
Jul 06 20:56:38.753108 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.753029    2569 scope.go:117] "RemoveContainer" containerID="7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"
Jul 06 20:56:38.753487 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:38.753464    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519\": container with ID starting with 7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519 not found: ID does not exist" containerID="7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"
Jul 06 20:56:38.753579 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.753497    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519"} err="failed to get container status \"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519\": rpc error: code = NotFound desc = could not find container \"7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519\": container with ID starting with 7a5fcdbb66c94af36b7d2777ab697df1caf75b50dedb5fde7a6e8ae5cc519519 not found: ID does not exist"
Jul 06 20:56:38.753579 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.753520    2569 scope.go:117] "RemoveContainer" containerID="22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"
Jul 06 20:56:38.753846 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:38.753823    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898\": container with ID starting with 22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898 not found: ID does not exist" containerID="22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"
Jul 06 20:56:38.753910 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.753855    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898"} err="failed to get container status \"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898\": rpc error: code = NotFound desc = could not find container \"22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898\": container with ID starting with 22b83a1548c8f306f3a5adc4f71a1c13c6493a90c773e72d2b750e7557334898 not found: ID does not exist"
Jul 06 20:56:38.761987 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.761956    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:56:38.768583 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:38.768556    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz"]
Jul 06 20:56:39.826732 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:39.826700    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" path="/var/lib/kubelet/pods/2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1/volumes"
Jul 06 20:56:45.752325 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:45.752298    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/0.log"
Jul 06 20:56:45.752704 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:45.752349    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerID="fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872" exitCode=1
Jul 06 20:56:45.752704 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:45.752377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerDied","Data":"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"}
Jul 06 20:56:46.757868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:46.757840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/0.log"
Jul 06 20:56:46.758284 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:46.757937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerStarted","Data":"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"}
Jul 06 20:56:47.858422 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:47.858380    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:47.858885 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:47.858657    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer" containerID="cri-o://af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6" gracePeriod=30
Jul 06 20:56:47.872550 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:47.872522    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:50.199299 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.199276    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/1.log"
Jul 06 20:56:50.199689 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.199673    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/0.log"
Jul 06 20:56:50.201868 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.201853    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:50.326024 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.325968    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert\") pod \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") "
Jul 06 20:56:50.326206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls\") pod \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") "
Jul 06 20:56:50.326206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326057    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config\") pod \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") "
Jul 06 20:56:50.326206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326126    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location\") pod \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") "
Jul 06 20:56:50.326206 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326149    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s422l\" (UniqueName: \"kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l\") pod \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\" (UID: \"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0\") "
Jul 06 20:56:50.326443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326378    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" (UID: "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:50.326443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326418    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" (UID: "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:50.326443 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.326436    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" (UID: "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:50.328303 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.328277    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l" (OuterVolumeSpecName: "kube-api-access-s422l") pod "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" (UID: "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0"). InnerVolumeSpecName "kube-api-access-s422l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:50.328398 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.328315    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" (UID: "0d1d570e-cd70-45f3-b7ea-1ff67e824ce0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:50.426904 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.426854    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:50.426904 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.426899    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s422l\" (UniqueName: \"kubernetes.io/projected/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-api-access-s422l\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:50.426904 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.426913    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:50.426904 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.426923    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:50.427194 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.426930    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:56:50.773875 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.773790    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/1.log"
Jul 06 20:56:50.774199 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774185    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2_0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/storage-initializer/0.log"
Jul 06 20:56:50.774260 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774219    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerID="af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6" exitCode=1
Jul 06 20:56:50.774306 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerDied","Data":"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"}
Jul 06 20:56:50.774342 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774301    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"
Jul 06 20:56:50.774384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2" event={"ID":"0d1d570e-cd70-45f3-b7ea-1ff67e824ce0","Type":"ContainerDied","Data":"678ef445f871f39e2560dff53e013676c239207c9a4da5649559c87bb1445c0c"}
Jul 06 20:56:50.774384 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.774379    2569 scope.go:117] "RemoveContainer" containerID="af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"
Jul 06 20:56:50.783060 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.783042    2569 scope.go:117] "RemoveContainer" containerID="fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"
Jul 06 20:56:50.790361 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.790344    2569 scope.go:117] "RemoveContainer" containerID="af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"
Jul 06 20:56:50.790651 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:50.790629    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6\": container with ID starting with af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6 not found: ID does not exist" containerID="af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"
Jul 06 20:56:50.790703 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.790660    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6"} err="failed to get container status \"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6\": rpc error: code = NotFound desc = could not find container \"af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6\": container with ID starting with af5740e3d54fe8ff6b740fdbea15dcac625e55c2f7610df7582d4b597e4241d6 not found: ID does not exist"
Jul 06 20:56:50.790703 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.790678    2569 scope.go:117] "RemoveContainer" containerID="fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"
Jul 06 20:56:50.790903 ip-10-0-139-250 kubenswrapper[2569]: E0706 20:56:50.790886    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872\": container with ID starting with fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872 not found: ID does not exist" containerID="fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"
Jul 06 20:56:50.790945 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.790908    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872"} err="failed to get container status \"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872\": rpc error: code = NotFound desc = could not find container \"fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872\": container with ID starting with fb1a044779e57911a77ae14d3350b6c5883ad68313fff26c641a55b4df15b872 not found: ID does not exist"
Jul 06 20:56:50.818436 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.818404    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:50.823663 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:50.823626    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2"]
Jul 06 20:56:51.826629 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:51.826594    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" path="/var/lib/kubelet/pods/0d1d570e-cd70-45f3-b7ea-1ff67e824ce0/volumes"
Jul 06 20:56:53.002486 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.002459    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:53.046332 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046301    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:56:53.046571 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046558    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kube-rbac-proxy"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046573    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kube-rbac-proxy"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046585    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046591    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046598    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046603    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046615    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046620    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046626    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="storage-initializer"
Jul 06 20:56:53.046630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046631    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="storage-initializer"
Jul 06 20:56:53.046918 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046678    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.046918 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046685    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kube-rbac-proxy"
Jul 06 20:56:53.046918 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046692    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2badce2e-d8c6-408c-8b4a-e55b3f7d3dc1" containerName="kserve-container"
Jul 06 20:56:53.046918 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.046788    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1d570e-cd70-45f3-b7ea-1ff67e824ce0" containerName="storage-initializer"
Jul 06 20:56:53.051308 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.051287    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.054166 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.054142    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:56:53.054270 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.054142    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:56:53.054270 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.054216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ef99f-predictor-serving-cert\""
Jul 06 20:56:53.054364 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.054326    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ef99f-kube-rbac-proxy-sar-config\""
Jul 06 20:56:53.055183 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.055169    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:56:53.055233 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.055189    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:56:53.055233 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.055191    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8r29g\""
Jul 06 20:56:53.064042 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.063995    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:56:53.115986 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.115955    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:53.155086 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.155047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.155275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.155146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.155275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.155180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.155275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.155218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.155275 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.155248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjjc\" (UniqueName: \"kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256484 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256484 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256484 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256787 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjjc\" (UniqueName: \"kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.256993 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.256970    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.257180 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.257162    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.257236 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.257215    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.259119 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.259100    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.267353 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.267324    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjjc\" (UniqueName: \"kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc\") pod \"raw-sklearn-ef99f-predictor-77b85567df-trpsw\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") " pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.361121 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.361080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:53.497186 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.496977    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:56:53.499806 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:56:53.499776    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84831045_9e8c_4f6f_9e53_2fbb1b4739e6.slice/crio-948da9331c0001d6d7b545b14baa61e014f3e45d669e1d731e07d843a04e22a7 WatchSource:0}: Error finding container 948da9331c0001d6d7b545b14baa61e014f3e45d669e1d731e07d843a04e22a7: Status 404 returned error can't find the container with id 948da9331c0001d6d7b545b14baa61e014f3e45d669e1d731e07d843a04e22a7
Jul 06 20:56:53.788065 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.788002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerStarted","Data":"de2b55231c9ab8f4391b4472f7f7d6686d348690e4e9d002832946264ebfe122"}
Jul 06 20:56:53.788337 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:53.788099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerStarted","Data":"948da9331c0001d6d7b545b14baa61e014f3e45d669e1d731e07d843a04e22a7"}
Jul 06 20:56:57.804814 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:57.804779    2569 generic.go:356] "Generic (PLEG): container finished" podID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerID="de2b55231c9ab8f4391b4472f7f7d6686d348690e4e9d002832946264ebfe122" exitCode=0
Jul 06 20:56:57.805226 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:57.804852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerDied","Data":"de2b55231c9ab8f4391b4472f7f7d6686d348690e4e9d002832946264ebfe122"}
Jul 06 20:56:58.811630 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:58.811592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerStarted","Data":"c3df7cddcd52d2315b05e195c74cb529ff46f20e5c7798c613e87aa0eee75f61"}
Jul 06 20:56:58.812030 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:58.811667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerStarted","Data":"42bb2c8a96f5a0a5d275bef40e355b60e1a07848fa5f7f098a8798afb27e3a8d"}
Jul 06 20:56:58.812030 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:58.811866    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:58.837454 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:58.837403    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podStartSLOduration=2.249299021 podStartE2EDuration="5.837388801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.58808978s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:58.836271276 +0000 UTC m=+1313.566836762" watchObservedRunningTime="2026-07-06 20:56:58.837388801 +0000 UTC m=+1313.567954285"
Jul 06 20:56:59.817561 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:59.817524    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:56:59.818705 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:56:59.818680    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:00.821731 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:00.821695    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:05.828087 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:05.828065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:57:05.828643 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:05.828615    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:15.828830 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:15.828791    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:25.828563 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:25.828521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:35.829345 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:35.829266    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:40.639704 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:40.639671    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:45.829565 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:45.829523    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:57:55.828979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:57:55.828943    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:58:05.828702 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:05.828663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:58:15.829153 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:15.829115    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:58:18.370453 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:18.370415    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:23.230276 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:23.230242    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:58:23.230707 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:23.230514    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kube-rbac-proxy" containerID="cri-o://c3df7cddcd52d2315b05e195c74cb529ff46f20e5c7798c613e87aa0eee75f61" gracePeriod=30
Jul 06 20:58:23.230707 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:23.230511    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" containerID="cri-o://42bb2c8a96f5a0a5d275bef40e355b60e1a07848fa5f7f098a8798afb27e3a8d" gracePeriod=30
Jul 06 20:58:23.245376 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:23.245347    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:24.133990 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:24.133960    2569 generic.go:356] "Generic (PLEG): container finished" podID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerID="c3df7cddcd52d2315b05e195c74cb529ff46f20e5c7798c613e87aa0eee75f61" exitCode=2
Jul 06 20:58:24.134187 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:24.134040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerDied","Data":"c3df7cddcd52d2315b05e195c74cb529ff46f20e5c7798c613e87aa0eee75f61"}
Jul 06 20:58:25.822674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:25.822627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:58:25.828529 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:25.828499    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:58:28.150092 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.150064    2569 generic.go:356] "Generic (PLEG): container finished" podID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerID="42bb2c8a96f5a0a5d275bef40e355b60e1a07848fa5f7f098a8798afb27e3a8d" exitCode=0
Jul 06 20:58:28.150402 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.150135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerDied","Data":"42bb2c8a96f5a0a5d275bef40e355b60e1a07848fa5f7f098a8798afb27e3a8d"}
Jul 06 20:58:28.279689 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.279661    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:58:28.290979 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.290912    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:28.366551 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366517    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 20:58:28.366822 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366809    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container"
Jul 06 20:58:28.366866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366826    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container"
Jul 06 20:58:28.366866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366848    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="storage-initializer"
Jul 06 20:58:28.366866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366853    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="storage-initializer"
Jul 06 20:58:28.366866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366858    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kube-rbac-proxy"
Jul 06 20:58:28.366866 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366863    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kube-rbac-proxy"
Jul 06 20:58:28.367079 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kserve-container"
Jul 06 20:58:28.367079 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.366933    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" containerName="kube-rbac-proxy"
Jul 06 20:58:28.370173 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.370155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.372885 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.372858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-321a7-predictor-serving-cert\""
Jul 06 20:58:28.373224 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.373209    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-321a7-kube-rbac-proxy-sar-config\""
Jul 06 20:58:28.383634 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.383612    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 20:58:28.407078 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.407044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls\") pod \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") "
Jul 06 20:58:28.407300 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.407283    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config\") pod \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") "
Jul 06 20:58:28.407466 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.407453    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location\") pod \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") "
Jul 06 20:58:28.407577 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.407566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert\") pod \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") "
Jul 06 20:58:28.407674 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.407663    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvjjc\" (UniqueName: \"kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc\") pod \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\" (UID: \"84831045-9e8c-4f6f-9e53-2fbb1b4739e6\") "
Jul 06 20:58:28.408519 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.408487    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84831045-9e8c-4f6f-9e53-2fbb1b4739e6" (UID: "84831045-9e8c-4f6f-9e53-2fbb1b4739e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:58:28.408767 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.408744    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84831045-9e8c-4f6f-9e53-2fbb1b4739e6" (UID: "84831045-9e8c-4f6f-9e53-2fbb1b4739e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:58:28.408901 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.408812    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84831045-9e8c-4f6f-9e53-2fbb1b4739e6" (UID: "84831045-9e8c-4f6f-9e53-2fbb1b4739e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:28.410594 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.410568    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc" (OuterVolumeSpecName: "kube-api-access-bvjjc") pod "84831045-9e8c-4f6f-9e53-2fbb1b4739e6" (UID: "84831045-9e8c-4f6f-9e53-2fbb1b4739e6"). InnerVolumeSpecName "kube-api-access-bvjjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:58:28.411036 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.410990    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84831045-9e8c-4f6f-9e53-2fbb1b4739e6" (UID: "84831045-9e8c-4f6f-9e53-2fbb1b4739e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:58:28.471300 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.471263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:28.509090 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.509258 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.509258 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.509258 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6t9z\" (UniqueName: \"kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509302    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509314    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509324    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509333    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:58:28.509372 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.509341    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjjc\" (UniqueName: \"kubernetes.io/projected/84831045-9e8c-4f6f-9e53-2fbb1b4739e6-kube-api-access-bvjjc\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 20:58:28.609966 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.609871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.609966 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.609927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6t9z\" (UniqueName: \"kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.609966 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.609949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.610260 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.610083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.610260 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.610138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.610358 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.610296    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.610612 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.610589    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.610736 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.610693    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.612590 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.612573    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.621090 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.621070    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6t9z\" (UniqueName: \"kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z\") pod \"raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.680262 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.680216    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:28.812626 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:28.812597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 20:58:28.815199 ip-10-0-139-250 kubenswrapper[2569]: W0706 20:58:28.815170    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891b75ec_62b9_4505_8121_e7855b6b4ab0.slice/crio-a4ddeac532c0d3b9e75a03d972893b9ec8ce7059e685e4e267c5dddf5d811139 WatchSource:0}: Error finding container a4ddeac532c0d3b9e75a03d972893b9ec8ce7059e685e4e267c5dddf5d811139: Status 404 returned error can't find the container with id a4ddeac532c0d3b9e75a03d972893b9ec8ce7059e685e4e267c5dddf5d811139
Jul 06 20:58:29.155448 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.155358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerStarted","Data":"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"}
Jul 06 20:58:29.155448 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.155423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerStarted","Data":"a4ddeac532c0d3b9e75a03d972893b9ec8ce7059e685e4e267c5dddf5d811139"}
Jul 06 20:58:29.157344 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.157323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw" event={"ID":"84831045-9e8c-4f6f-9e53-2fbb1b4739e6","Type":"ContainerDied","Data":"948da9331c0001d6d7b545b14baa61e014f3e45d669e1d731e07d843a04e22a7"}
Jul 06 20:58:29.157473 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.157385    2569 scope.go:117] "RemoveContainer" containerID="c3df7cddcd52d2315b05e195c74cb529ff46f20e5c7798c613e87aa0eee75f61"
Jul 06 20:58:29.157473 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.157391    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"
Jul 06 20:58:29.166597 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.166579    2569 scope.go:117] "RemoveContainer" containerID="42bb2c8a96f5a0a5d275bef40e355b60e1a07848fa5f7f098a8798afb27e3a8d"
Jul 06 20:58:29.173738 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.173721    2569 scope.go:117] "RemoveContainer" containerID="de2b55231c9ab8f4391b4472f7f7d6686d348690e4e9d002832946264ebfe122"
Jul 06 20:58:29.193618 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.193587    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:58:29.199817 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.199791    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw"]
Jul 06 20:58:29.826391 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:29.826358    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84831045-9e8c-4f6f-9e53-2fbb1b4739e6" path="/var/lib/kubelet/pods/84831045-9e8c-4f6f-9e53-2fbb1b4739e6/volumes"
Jul 06 20:58:33.174318 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:33.174273    2569 generic.go:356] "Generic (PLEG): container finished" podID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerID="38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f" exitCode=0
Jul 06 20:58:33.174679 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:33.174344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerDied","Data":"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"}
Jul 06 20:58:34.179366 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:34.179327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerStarted","Data":"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"}
Jul 06 20:58:34.179862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:34.179402    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerStarted","Data":"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"}
Jul 06 20:58:34.179862 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:34.179612    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:34.206163 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:34.206112    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podStartSLOduration=2.76651074 podStartE2EDuration="6.206098148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.439587408s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:34.202775919 +0000 UTC m=+1408.933341404" watchObservedRunningTime="2026-07-06 20:58:34.206098148 +0000 UTC m=+1408.936663679"
Jul 06 20:58:35.183580 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:35.183551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:35.184938 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:35.184911    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:58:36.187544 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:36.187499    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:58:41.192436 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:41.192409    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:58:41.192955 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:41.192929    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:58:51.193034 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:58:51.192984    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:01.193338 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:01.193257    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:11.193486 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:11.193439    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:21.193821 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:21.193782    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:31.193291 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:31.193240    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:41.193416 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:41.193377    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:48.823170 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:48.823138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 20:59:58.512690 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:58.512653    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 20:59:58.513254 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:58.512925    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" containerID="cri-o://deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99" gracePeriod=30
Jul 06 20:59:58.513254 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:58.512970    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kube-rbac-proxy" containerID="cri-o://39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd" gracePeriod=30
Jul 06 20:59:58.547480 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:58.547441    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:58.822602 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:58.822570    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:59:59.496720 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:59.496677    2569 generic.go:356] "Generic (PLEG): container finished" podID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerID="39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd" exitCode=2
Jul 06 20:59:59.496900 ip-10-0-139-250 kubenswrapper[2569]: I0706 20:59:59.496736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerDied","Data":"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"}
Jul 06 21:00:01.187999 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:01.187951    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 06 21:00:03.762853 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.762825    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 21:00:03.880869 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.880774    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config\") pod \"891b75ec-62b9-4505-8121-e7855b6b4ab0\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") "
Jul 06 21:00:03.880869 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.880819    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls\") pod \"891b75ec-62b9-4505-8121-e7855b6b4ab0\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") "
Jul 06 21:00:03.880869 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.880861    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location\") pod \"891b75ec-62b9-4505-8121-e7855b6b4ab0\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") "
Jul 06 21:00:03.881177 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.880887    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6t9z\" (UniqueName: \"kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z\") pod \"891b75ec-62b9-4505-8121-e7855b6b4ab0\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") "
Jul 06 21:00:03.881177 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.880925    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert\") pod \"891b75ec-62b9-4505-8121-e7855b6b4ab0\" (UID: \"891b75ec-62b9-4505-8121-e7855b6b4ab0\") "
Jul 06 21:00:03.881331 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.881299    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "891b75ec-62b9-4505-8121-e7855b6b4ab0" (UID: "891b75ec-62b9-4505-8121-e7855b6b4ab0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:00:03.881331 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.881314    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "891b75ec-62b9-4505-8121-e7855b6b4ab0" (UID: "891b75ec-62b9-4505-8121-e7855b6b4ab0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:00:03.881443 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.881388    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "891b75ec-62b9-4505-8121-e7855b6b4ab0" (UID: "891b75ec-62b9-4505-8121-e7855b6b4ab0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:00:03.883258 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.883229    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z" (OuterVolumeSpecName: "kube-api-access-g6t9z") pod "891b75ec-62b9-4505-8121-e7855b6b4ab0" (UID: "891b75ec-62b9-4505-8121-e7855b6b4ab0"). InnerVolumeSpecName "kube-api-access-g6t9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:00:03.883374 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.883270    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "891b75ec-62b9-4505-8121-e7855b6b4ab0" (UID: "891b75ec-62b9-4505-8121-e7855b6b4ab0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:00:03.982266 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.982223    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6t9z\" (UniqueName: \"kubernetes.io/projected/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-api-access-g6t9z\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 21:00:03.982266 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.982264    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-cabundle-cert\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 21:00:03.982266 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.982274    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/891b75ec-62b9-4505-8121-e7855b6b4ab0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 21:00:03.982508 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.982283    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/891b75ec-62b9-4505-8121-e7855b6b4ab0-proxy-tls\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 21:00:03.982508 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:03.982292    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/891b75ec-62b9-4505-8121-e7855b6b4ab0-kserve-provision-location\") on node \"ip-10-0-139-250.ec2.internal\" DevicePath \"\""
Jul 06 21:00:04.514708 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.514666    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:04.516775 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.516751    2569 generic.go:356] "Generic (PLEG): container finished" podID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerID="deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99" exitCode=0
Jul 06 21:00:04.516905 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.516806    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"
Jul 06 21:00:04.516905 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.516805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerDied","Data":"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"}
Jul 06 21:00:04.516981 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.516948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m" event={"ID":"891b75ec-62b9-4505-8121-e7855b6b4ab0","Type":"ContainerDied","Data":"a4ddeac532c0d3b9e75a03d972893b9ec8ce7059e685e4e267c5dddf5d811139"}
Jul 06 21:00:04.517038 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.516990    2569 scope.go:117] "RemoveContainer" containerID="39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"
Jul 06 21:00:04.528860 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.528834    2569 scope.go:117] "RemoveContainer" containerID="deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"
Jul 06 21:00:04.537045 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.537021    2569 scope.go:117] "RemoveContainer" containerID="38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"
Jul 06 21:00:04.545335 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.545313    2569 scope.go:117] "RemoveContainer" containerID="39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"
Jul 06 21:00:04.545646 ip-10-0-139-250 kubenswrapper[2569]: E0706 21:00:04.545624    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd\": container with ID starting with 39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd not found: ID does not exist" containerID="39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"
Jul 06 21:00:04.545741 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.545658    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd"} err="failed to get container status \"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd\": rpc error: code = NotFound desc = could not find container \"39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd\": container with ID starting with 39f4104dec22904c97a0d050a4bb7e0b48d1e04a1a6ce4e7363a4c316ed9dbfd not found: ID does not exist"
Jul 06 21:00:04.545741 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.545684    2569 scope.go:117] "RemoveContainer" containerID="deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"
Jul 06 21:00:04.545978 ip-10-0-139-250 kubenswrapper[2569]: E0706 21:00:04.545959    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99\": container with ID starting with deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99 not found: ID does not exist" containerID="deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"
Jul 06 21:00:04.546043 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.545983    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99"} err="failed to get container status \"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99\": rpc error: code = NotFound desc = could not find container \"deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99\": container with ID starting with deb624af18a75a7bd3e31f2db859f13169a452629322d5b17cb778a1a556db99 not found: ID does not exist"
Jul 06 21:00:04.546043 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.546000    2569 scope.go:117] "RemoveContainer" containerID="38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"
Jul 06 21:00:04.546303 ip-10-0-139-250 kubenswrapper[2569]: E0706 21:00:04.546285    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f\": container with ID starting with 38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f not found: ID does not exist" containerID="38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"
Jul 06 21:00:04.546361 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.546307    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f"} err="failed to get container status \"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f\": rpc error: code = NotFound desc = could not find container \"38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f\": container with ID starting with 38159d60a949f9b56ab8d5d7dc27b328cd6f599085677b6d659242a27922d36f not found: ID does not exist"
Jul 06 21:00:04.546452 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.546434    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 21:00:04.552134 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.552109    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m"]
Jul 06 21:00:04.576434 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:04.576405    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:05.827098 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:05.827062    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" path="/var/lib/kubelet/pods/891b75ec-62b9-4505-8121-e7855b6b4ab0/volumes"
Jul 06 21:00:34.991879 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.991803    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6q7z/must-gather-rqx5s"]
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992142    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kube-rbac-proxy"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992155    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kube-rbac-proxy"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992178    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="storage-initializer"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992184    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="storage-initializer"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992191    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992196    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992241    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kube-rbac-proxy"
Jul 06 21:00:34.994070 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.992249    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="891b75ec-62b9-4505-8121-e7855b6b4ab0" containerName="kserve-container"
Jul 06 21:00:34.994794 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.994780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:34.997495 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.997476    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6q7z\"/\"default-dockercfg-qrp5b\""
Jul 06 21:00:34.997724 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.997709    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6q7z\"/\"openshift-service-ca.crt\""
Jul 06 21:00:34.997771 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:34.997712    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6q7z\"/\"kube-root-ca.crt\""
Jul 06 21:00:35.004304 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.004279    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6q7z/must-gather-rqx5s"]
Jul 06 21:00:35.113282 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.113248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74bs\" (UniqueName: \"kubernetes.io/projected/887df5ad-37b4-4755-8921-71fe3200f168-kube-api-access-k74bs\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.113463 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.113306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/887df5ad-37b4-4755-8921-71fe3200f168-must-gather-output\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.213928 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.213892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74bs\" (UniqueName: \"kubernetes.io/projected/887df5ad-37b4-4755-8921-71fe3200f168-kube-api-access-k74bs\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.214155 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.213985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/887df5ad-37b4-4755-8921-71fe3200f168-must-gather-output\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.214335 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.214317    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/887df5ad-37b4-4755-8921-71fe3200f168-must-gather-output\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.225111 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.225080    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74bs\" (UniqueName: \"kubernetes.io/projected/887df5ad-37b4-4755-8921-71fe3200f168-kube-api-access-k74bs\") pod \"must-gather-rqx5s\" (UID: \"887df5ad-37b4-4755-8921-71fe3200f168\") " pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.303882 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.303851    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6q7z/must-gather-rqx5s"
Jul 06 21:00:35.450635 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.450602    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6q7z/must-gather-rqx5s"]
Jul 06 21:00:35.453664 ip-10-0-139-250 kubenswrapper[2569]: W0706 21:00:35.453636    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887df5ad_37b4_4755_8921_71fe3200f168.slice/crio-5e1a6406cd2fd53b34090ea7ef3c19e4d66f7e5e4dc5fe9fccd9b0e0ba3cee60 WatchSource:0}: Error finding container 5e1a6406cd2fd53b34090ea7ef3c19e4d66f7e5e4dc5fe9fccd9b0e0ba3cee60: Status 404 returned error can't find the container with id 5e1a6406cd2fd53b34090ea7ef3c19e4d66f7e5e4dc5fe9fccd9b0e0ba3cee60
Jul 06 21:00:35.455530 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.455512    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:00:35.577139 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.577059    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:35.638391 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.638353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6q7z/must-gather-rqx5s" event={"ID":"887df5ad-37b4-4755-8921-71fe3200f168","Type":"ContainerStarted","Data":"5e1a6406cd2fd53b34090ea7ef3c19e4d66f7e5e4dc5fe9fccd9b0e0ba3cee60"}
Jul 06 21:00:35.652793 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:35.652764    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:36.645868 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:36.645818    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6q7z/must-gather-rqx5s" event={"ID":"887df5ad-37b4-4755-8921-71fe3200f168","Type":"ContainerStarted","Data":"9f26bab134d4c07a4f0d193100c969e8caec0812f442b09de5f104682ff9f9a7"}
Jul 06 21:00:36.646316 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:36.645882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6q7z/must-gather-rqx5s" event={"ID":"887df5ad-37b4-4755-8921-71fe3200f168","Type":"ContainerStarted","Data":"04749e5aaa5101de55099e056f847dbfa92dbed6d1e012fec497ad50c602ed56"}
Jul 06 21:00:36.669312 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:36.669237    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6q7z/must-gather-rqx5s" podStartSLOduration=1.8471608210000001 podStartE2EDuration="2.669213577s" totalImagesPullingTime="822.052756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:36.667967844 +0000 UTC m=+1531.398533330" watchObservedRunningTime="2026-07-06 21:00:36.669213577 +0000 UTC m=+1531.399779064"
Jul 06 21:00:37.953786 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:37.953753    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-flcrl_5d01ff7c-1c2f-4b03-94b9-fcf6b5630b24/global-pull-secret-syncer/0.log"
Jul 06 21:00:38.059290 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:38.059255    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log"
Jul 06 21:00:38.220840 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:38.220754    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-250.ec2.internal_91570a53ecc3c9d70fa440a335fdf4cb/haproxy/0.log"
Jul 06 21:00:41.170438 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:41.170400    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:41.228730 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:41.228664    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7bwl2_89105b5b-bd31-4936-8f37-f491da14518a/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:42.356983 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.356866    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xpndf_87254369-84ea-43f1-9c45-6267c77a1997/kube-state-metrics/0.log"
Jul 06 21:00:42.389717 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.389684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xpndf_87254369-84ea-43f1-9c45-6267c77a1997/kube-rbac-proxy-main/0.log"
Jul 06 21:00:42.432663 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.432630    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xpndf_87254369-84ea-43f1-9c45-6267c77a1997/kube-rbac-proxy-self/0.log"
Jul 06 21:00:42.506731 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.506705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-db5ks_7c0e104c-11cd-411b-9362-3358b8083b59/monitoring-plugin/0.log"
Jul 06 21:00:42.773960 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.773867    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsnn_ac677c71-6a1c-4e48-b82a-8d6c4dd18913/node-exporter/0.log"
Jul 06 21:00:42.803286 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.803256    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsnn_ac677c71-6a1c-4e48-b82a-8d6c4dd18913/kube-rbac-proxy/0.log"
Jul 06 21:00:42.834483 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:42.834456    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsnn_ac677c71-6a1c-4e48-b82a-8d6c4dd18913/init-textfile/0.log"
Jul 06 21:00:43.273106 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:43.273077    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-nkv2t_5979f59f-d2f2-4d79-ba3f-47a3100087b4/prometheus-operator-admission-webhook/0.log"
Jul 06 21:00:45.041461 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:45.041427    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-6m5br_1ad96b72-edd0-4f3d-a5f5-50a0b499493d/networking-console-plugin/0.log"
Jul 06 21:00:46.012832 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.012795    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"]
Jul 06 21:00:46.018863 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.018832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.048027 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.047972    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"]
Jul 06 21:00:46.117059 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.116989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-podres\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.117264 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.117089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9wr\" (UniqueName: \"kubernetes.io/projected/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-kube-api-access-ng9wr\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.117264 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.117116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-sys\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.117264 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.117166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-lib-modules\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.117402 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.117258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-proc\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218240 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-podres\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9wr\" (UniqueName: \"kubernetes.io/projected/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-kube-api-access-ng9wr\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-sys\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-lib-modules\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218386    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-proc\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218390    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-podres\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218444 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218426    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-sys\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218797 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218471    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-proc\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.218797 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.218511    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-lib-modules\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.229560 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.229532    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9wr\" (UniqueName: \"kubernetes.io/projected/c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54-kube-api-access-ng9wr\") pod \"perf-node-gather-daemonset-nfkgx\" (UID: \"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54\") " pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.329400 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.329370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.405548 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.405431    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-8fxx5_ef7d8d3e-783e-4880-b9f9-c1c9ede9e5d6/download-server/0.log"
Jul 06 21:00:46.511469 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.511382    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"]
Jul 06 21:00:46.699376 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.699334    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx" event={"ID":"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54","Type":"ContainerStarted","Data":"77e0212e4c458924a940c9094d0bc91e768ebe7d742659cd5c9d0b5ea345eaf0"}
Jul 06 21:00:46.699572 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.699403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx" event={"ID":"c46d9eb9-1cbc-4df7-a7d5-d12c1a95dd54","Type":"ContainerStarted","Data":"76696322ed2ee5ec6c0bfe931416a6aa1fb42d4a87dde6afd15a05eb9e847b6b"}
Jul 06 21:00:46.699572 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.699516    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:46.721527 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:46.721475    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx" podStartSLOduration=1.721461321 podStartE2EDuration="1.721461321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:46.720921711 +0000 UTC m=+1541.451487198" watchObservedRunningTime="2026-07-06 21:00:46.721461321 +0000 UTC m=+1541.452026807"
Jul 06 21:00:48.177659 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:48.177610    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j9cl_2abd5cc4-49a1-4779-8895-bd4e75c7e24b/dns/0.log"
Jul 06 21:00:48.210762 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:48.210729    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j9cl_2abd5cc4-49a1-4779-8895-bd4e75c7e24b/kube-rbac-proxy/0.log"
Jul 06 21:00:48.469270 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:48.469194    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgq64_f73822cb-17b8-4dde-b27f-dff1507bba12/dns-node-resolver/0.log"
Jul 06 21:00:49.136032 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:49.135988    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvvlz_432ed208-35eb-45fb-801c-f1fddff9a59a/node-ca/0.log"
Jul 06 21:00:50.156306 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:50.156280    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-98bb9bc6f-bfglc_ca8fbc18-473f-4e4c-bffd-a44971c092ea/router/0.log"
Jul 06 21:00:50.673414 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:50.673374    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjv6r_77c9edc8-8c55-4d7e-9639-143e3e6aca7c/serve-healthcheck-canary/0.log"
Jul 06 21:00:51.135518 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:51.135487    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-746tn_07b86944-4f5d-4b78-8102-834bf1989fcd/insights-operator/0.log"
Jul 06 21:00:51.135727 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:51.135708    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-746tn_07b86944-4f5d-4b78-8102-834bf1989fcd/insights-operator/1.log"
Jul 06 21:00:51.167234 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:51.167210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrfhw_18e46866-7fc7-4c1b-99c0-fa620cd9093f/kube-rbac-proxy/0.log"
Jul 06 21:00:51.202069 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:51.202034    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrfhw_18e46866-7fc7-4c1b-99c0-fa620cd9093f/exporter/0.log"
Jul 06 21:00:51.233708 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:51.233682    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrfhw_18e46866-7fc7-4c1b-99c0-fa620cd9093f/extractor/0.log"
Jul 06 21:00:52.714653 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:52.714626    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6q7z/perf-node-gather-daemonset-nfkgx"
Jul 06 21:00:53.680292 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:00:53.680260    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68bc99fcb8-597z5_51255cbe-5c11-4cd8-a95f-efef70249cec/manager/0.log"
Jul 06 21:01:00.424817 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.424779    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/kube-multus-additional-cni-plugins/0.log"
Jul 06 21:01:00.455767 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.455736    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/egress-router-binary-copy/0.log"
Jul 06 21:01:00.484168 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.484142    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/cni-plugins/0.log"
Jul 06 21:01:00.515426 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.515397    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/bond-cni-plugin/0.log"
Jul 06 21:01:00.544907 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.544879    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/routeoverride-cni/0.log"
Jul 06 21:01:00.573935 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.573906    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/whereabouts-cni-bincopy/0.log"
Jul 06 21:01:00.608555 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.608525    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xg5rc_8f689c71-25d4-4f64-ba0a-2da7907bf33b/whereabouts-cni/0.log"
Jul 06 21:01:00.961253 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:00.961225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jp5cv_1470ecd4-28e4-45c5-82f2-78a68a38f2d5/kube-multus/0.log"
Jul 06 21:01:01.057244 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:01.057210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wbpj_8f6da2d8-84da-4bd9-b268-4414bbdce3b1/network-metrics-daemon/0.log"
Jul 06 21:01:01.085210 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:01.085178    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wbpj_8f6da2d8-84da-4bd9-b268-4414bbdce3b1/kube-rbac-proxy/0.log"
Jul 06 21:01:02.350812 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.350785    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/ovn-controller/0.log"
Jul 06 21:01:02.379052 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.379020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/ovn-acl-logging/0.log"
Jul 06 21:01:02.409382 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.409348    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/kube-rbac-proxy-node/0.log"
Jul 06 21:01:02.436902 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.436878    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/kube-rbac-proxy-ovn-metrics/0.log"
Jul 06 21:01:02.461296 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.461270    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/northd/0.log"
Jul 06 21:01:02.489214 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.489190    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/nbdb/0.log"
Jul 06 21:01:02.517527 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.517499    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/sbdb/0.log"
Jul 06 21:01:02.568543 ip-10-0-139-250 kubenswrapper[2569]: I0706 21:01:02.568512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ckcf_9c788a21-75b3-47f8-ac61-57014732d767/ovnkube-controller/0.log"